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SUMMARY  AND  CONCLUSIONS 
EVALUATION  OF  AUTOMATED  ECG  SERVICES 

I .  SUMMARY 

el  Under  sponsorship  of  the  Tri-Service  Medical  Information  Systems 
(TRIMIS)  Program  Office  (TPO) ,  a  computer  system  for  analysis  of  electro¬ 
cardiograms  (ECGs)  was  installed  at  18  Medical  Treatment  Facilities  (MTFs)  In 
the  Southern  California  area.  The  objective  of  the  automated  ECG  system  is  to 
improve  cardiac  diagnosis  and  treatment  in  the  military  health  care  system 
by  providing  high-quality  ECG  analysis  services,  at  a  reasonable  cost,  to 
a  large  number  of  MTFs,  many  of  which  do  not  have  cardiologists  on  their 
staffs . 

'  >  r  / 

Electrocardiograms  are  used  to  evaluate  the  cardiac  status  of  a 
patient.  The  ECG  can  be  taken  inexpensively  and  with  no  risk  to  the 
patient  (because  the  procedure  is  non- invasive) .  Therefore,  the  ECG  is 
a  basic  tool  in  performing  a  cardiac  evaluation.  The  interpretation  of 
an  electrocardiogram  is  usually  combined  with  other  test  results  and  the 
findings  of  physical  examination  in  order  to  assess  the  patient’s  cardiac 
status. 

In  the  military,  ECGs  are  used  to  screen  healthy  Individuals  to  ensure 
that  they  are  free  of  cardiac  abnormalities,  as  a  physical  examination 
and  other  tests  have  suggested.  For  example,  the  ECG  is  a  standard  com¬ 
ponent  of  flight  physical  examinations.  An  ECG  is  also  used  as  a  diag¬ 
nostic  tool.  For  instance,  it  is  one  of  several  tests  used  to  evaluate 
whether  a  patient  with  chest  pain  has  had  a  heart  attack.  It  is  also 
used  to  monitor  progress  in  patients  with  known  cardiac  disease,  such  as 
in  determining  1 r  ’  heart  attack  is  resolving  or  if  a  new  medication  is 
effectively  trua  "  ^  cardiac  problem. 

Electrocarr  _  a  for  these  various  purposes  are  taken  in  all  MTFs. 

Since  one  of  the  r-  .  ases  of  ECGs  is  the  diagnosis  and  treatment  of 
patients  that  are  potentially  critically  ill,  ECG  interpretations  must 
also  be  available  in  all  MTFs.  However,  trained  cardiologists  are  not 
available  within  the  military  to  staff  every  MTF,  and  in  some  sites,  no 
specialists  in  internal  medicine  are  assigned  either.  Since  these  facilities 
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are  often  remote,  civilian  specialists  may  not  be  readily  available. 

The  automated  ECG  system,  which  links  remote  facilities  with  large 
hospitals  that  have  specially- trained  physicians  on  the  staff  and  provides 
rapid  access  to  cardiologist  consultation,  was  installed  to  make  ECG 
interpretations  more  readily  available  at  these  facilities. 

The  costs  and  effectiveness  of  ECG  services  have  been  evaluated  in 
the  18  project  sites  before  and  after  the  automation  of  ECG  services. 

The  conclusion  of  the  evaluation  is  that  an  automated  system  provides 
more  cost-effective  services  than  can  be  provided  by  manual  methods. 

The  major  benefits  of  automation  are  related  to  access  to  and  effi¬ 
ciency  and  quality  of  ECG  services: 

•  improved  access  at  small  medical  treatment  facilities  to 
cardiologists , 

•  support  to  non-cardiologist  physicians  in  reading  electro¬ 
cardiograms,  and 

e  decreased  time  to  receive  an  ECG  interpretation  at  remote 
treatment  facilities. 

Other  benefits  include: 

•  increased  availability  of  prior  electrocardiograms  for 
comparison  with  the  current  ECG, 

•  saved  time  in  reading  electrocardiograms, 

e  Improved  technical  quality  of  the  ECG  tracing, 

•  increased  legibility  and  consistency  of  the  ECG  report, 

•  fewer  lost  ECG  tracings  and  reports,  and 

•  Improved  accuracy  of  reading  (especially  for  normal  tracings). 

The  effectiveness  of  ECG  services  before  and  after  automation  was 
measured  against  service  criteria  related  to  the  availability  of  ECG 
readers  with  specialized  training,  the  time  delay  before  receipt  of  ECG 
interpretations,  and  the  availability  of  prior  ECGs  for  comparison  with 
a  current  ECG.  When  the  computer  system  is  fully  operational,  the 
portion  of  ECGs  meeting  effectiveness  criteria  in  the  smaller  MFTs  will 
have  increased  from  38  percent  to  86  percent;  in  the  larger  MTFs  effective 
ECG  services  will  have  increased  from  56  percent  of  all  ECGs  to  75  percent. 
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The  costs  of  providing  ECG  services  in  the  18  MTFs  were  analyzed 
for  the  manual  ECG  services  and  the  automated  system.  These  costs  were 
divided  by  the  number  of  ECGs  meeting  criteria  for  effective  ECG  services 
in  order  to  calculate  a  cost  per  effective  ECG.  The  cost  per  effective 
ECG  decreases  from  $15  to  $12  when  the  system  is  fully  operational;  the 
cost  per  ECG  meeting  preferred  service  criteria  will  decrease  from 
$107  to  $18. 

Thus,  automated  ECG  services  do  meet  the  objective  of  providing 
high-quality  ECG  services  in  all  Medical  Treatment  Facilities. 

II.  INTRODUCTION 


The  computer  system,  which  was  the  subject  of  this  evaluation,  is 
referred  to  as  CAPOC  (Computer  Assisted  Practice  of  Cardiology).  It  has 
one  central  computer  (located  at  San  Diego  Naval  Regional  Medical  Center) , 
which  is  used  to  provide  ECG  interpretations  for  MTFs  within  the  Southern 
California  area.  From  remote  sites,  electrocardiograms  (ECGs)  are  trans¬ 
mitted  over  telephone  lines  to  the  central  system  and  an  interpretation 
is  performed  by  the  computer,  transmitted  over  the  telephone  lines,  and 
printed  out  within  about  15  minutes  at  the  site  where  the  ECG  was  taken. 
The  computer-generated  interpretations  can  also  be  confirmed  by  cardiolo¬ 
gists  at  the  central  site  and  then  returned. 

The  evaluation  of  the  CAPOC  system  was  designed  to  provide  informa¬ 
tion  for  decisionmaking  regarding  purchase  of  additional  systems  for 
installation  in  other  regions,  to  determine  whether  or  not  the  system 
met  its  objective  of  providing  high-quality,  cost-effective  ECG  services, 
especially  to  small  facilities  without  a  cardiologist  on  the  staff,  and 
to  identify  changes  that  should  be  made  to  increase  the  effectiveness  of 
the  system,  or  lower  its  cost. 

In  this  summary  section,  the  methodology  developed  for  use  in  the 
evaluation  will  be  described  and  then  information  on  the  findings  of  the 
evaluation  will  be  discussed.  More  comprehensive  information  on  the 
evaluation  approach,  analysis  techniques,  and  results  of  the  study  are 
contained  in  the  following  chapters  of  this  final  report. 
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III.  EVALUATION  APPROACH 


The  evaluation  was  designed  to  compare  manual  ECG  services  before 
automation  with  services  provided  after  the  CAPOC  system  had  been  imple¬ 
mented  . 

In  order  to  determine  whether  the  CAPOC  system  did  improve  ECG 
services,  a  definition  of  effective  ECG  services  was  required.  This  was 
accomplished  by  a  consensus  process  involving  ECG  users  in  MTFs.  Survey 
responses  were  obtained  from  55  physicians,  and  the  results  were  reviewed 
by  a  panel  of  nine  internists,  cardiologists,  and  ECG  technicians,  who 
set  the  final  criteria. 

The  users  who  participated  in  the  consensus  process  identified  three 
essential  attributes  of  effective  ECG  services: 

(1)  accuracy  of  the  interpretation, 

(2)  turnaround  time  to  receive  the  interpretation,  and 

(3)  use  of  serial  comparison  with  prior  ECGs. 

Based  upon  these  attributes,  they  defined  two  levels  of  effective  ECG 
services:  what  is  typically  required  for  the  ECG  to  be  useful  in  clini¬ 
cal  management  of  patients  (acceptable  service)  and  what  permits  the 
maximum  benefit  from  ECG  information  in  clinical  decisionmaking  (pre¬ 
ferred  service).  Together  these  service  levels  constitute  what  we  will 
refer  to  as  "effective"  ECG  services.  The  criteria  for  effective  ECG 
services  that  were  developed  and  used  in  this  evaluation  are  shown  in 
Table  S-l. 

The  "quality"  of  ECG  services,  as  measured  against  these  criteria, 
was  quantified  for  manual  and  automated  ECG  services.  As  an  example, 
the  criteria  for  turnaround  time  for  routine  inpatient  ECGs  are  that: 
an  ECG  returned  within  2  days  is  acceptable,  and  an  ECG  returned  in 

1  day  is  preferred.  Therefore,  routine  inpatient  ECGs  not  returned  within 

2  days  were  not  counted  as  "effective"  ECGs  in  the  evaluation.  ECGs  re¬ 
turned  within  2  days  were  counted  as  effective  (if  other  service  criteria 
were  also  met).  If  the  ECG  was  returned  within  1  day,  it  was  counted 

as  an  ECG  that  met  the  preferred  level  of  service. 
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TABLE  S-l 


EVALUATION  CRITERIA  FOR  ECG  SERVICES 


Criteria  for  Acceptable  Services 

•  In  a  manual  reading  system  ECGs  are  read  by  a  cardiologist  or 
internist. 

•  In  a  computer-assisted  reading  system,  abnormal  ECG  interpretations 
are  reviewed  by  a  cardiologist  or  internist;  normal  interpretations 
are  reviewed  by  a  corpsman,  general  pratitioner,  internist,  or 
cardiologist. 

•  ECG  interpretations  are  acceptable  if  returned  within  the  following 
times: 

-  Routine  Inpatient:  2  days 

-  Pre-Operative:  1  day 

-  Emergency:  30  minutes 

-  Routine  Outpatient:  3  days 

-  Annual  Physical:  7  days 

•  All  abnormal  ECGs  receive  a  serial  comparison 


Criteria  for  Preferred  Services 


•  In  a  manual  reading  system  ECGs  are  read  by  a  cardiologist; 

•  In  a  computer-assisted  reading  system,  abnormal  ECG  interpretations 
are  reviewed  by  a  cardiologist;  normal  interpretations  are  reviewed 
by  a  corpsman,  general  practitioner,  internist  or  cardiologist. 

•  ECG  interpretations  are  preferred  if  returned  within  the  following 
times: 

-  Routine  Inpatient:  1  day 

-  Pre-Operative:  12  hours 

-  Emergency:  15  minutes 

-  Routine  Outpatient:  30  minutes 

-  Annual  Physical :  2  days 

•  All  abnormal  ECGs  receive  a  serial  comparison 
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The  cost  of  providing  manual  and  CAPOC  services  was  also  computed 
and  the  cost  to  provide  an  ECG  meeting  the  criteria  for  effective  services 
was  calculated.  If  the  cost  per  effective  ECG  was  lower  with  CAPOC, 
the  system  would  be  considered  successful. 

The  data  on  baseline  (manual)  services  were  collected  in  the  Fall 
of  1979.  Data  on  CAPOC  services  were  obtained  in  the  Fall  of  1980.  To 
limit  the  cost  of  data  collection,  five  sites  were  chosen  as  primary 
evaluation  sites.  These  included  the  largest  referral  site  in  the  region, 
a  smaller  referral  hospital,  a  small  branch  clinic,  a  small  remote  branch 
hospital,  and  one  small  hospital  not  affiliated  with  a  large  referral 
center.  The  sites  were  chosen  to  be  representative  of  the  staffing  pat¬ 
terns,  patient  populations,  and  ECG  services  found  in  the  region.  Infor¬ 
mation  on  ECG  services  was  collected  at  the  primary  evaluation  sites 
through  on-site  data  collection,  review  of  hospital  and  ECG  department 
records,  interviews  with  staff,  and  administration  of  a  survey  to  ECG 
readers  and  users  of  ECG  services. 

Brief  site  visits  were  conducted  to  the  remaining  13  secondary 
evaluation  sites  in  the  region  in  order  to  obtain  information  on  manual 
and  automated  ECG  services.  In  these  site  visits  a  description  of  the 
ECG  process  was  obtained  and  estimates  or  data  were  obtained  on  the  per¬ 
centage  of  normal  and  abnormal  ECGs  processed,  the  turnaround  time  to 
receive  an  interpretation,  and  the  training  of  ECG  readers. 

These  data  were  then  used  to  estimate  effectiveness  of  services  in 
the  region  with  both  manual  and  automated  services  and  to  determine  the 
impact  of  CAPOC  on  the  costs  and  effectiveness  of  services. 

IV.  EVALUATION  FINDINGS 
A.  Summary  Findings 

The  major  finding  of  the  CAPOC  evaluation  is  that  automated  ECG  ser¬ 
vices  provide  high-quality,  cost-effective  ECG  services  within  the  DoD  health 
care  system.  In  the  Southern  California  area,  the  cost  per  effective  ECG 
will  decrease  from  $15  to  $12  when  CAPOC  is  fully  operational.  The  cost 
per  ECG  meeting  the  preferred  service  levels  will  drop  from  $107  to  $18. 

The  greatest  benefits  from  automation  are  realized  at  small  MTFs. 
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For  these  sites,  the  CAPOC  system: 

•  provides  increased  access  to  cardiologists,  located  primarily 
at  large  MTFs.  (These  cardiologists  can  either  routinely 
review  tracings  or  be  asked  to  interpret  a  specific  ECG.) 

•  supports  non-cardiologists  in  reading  normal  electrocardio¬ 
grams  by  providing  a  second  opinion — the  computer  interpre¬ 
tation,  and 

•  makes  ECG  interpretations  available  from  outside  the 
facility  in  minutes  rather  than  days. 

At  the  larger  MTFs ,  CAPOC : 

•  increases  access  to  prior  electrocardiograms  (which  can 

be  stored  in  the  computer  and  printed  out  automatically),  and 

•  reduces  cardiologists'  time  required  to  interpret  the 
electrocardiogram. 

At  all  MTFs,  CAPOC: 

•  improves  the  quality  of  ECG  tracings,  resulting  in  fewer 
repeat  tracings  or  fewer  ECGs  that  cannot  be  interpreted, 

•  increases  the  ease  of  reading  and  understanding  interpre¬ 
tations  (since  they  are  printed  rather  than  handwritten) , 
and 

•  decreases  the  number  of  ECG  tracings  and  interpretations 
lost,  resulting  in  fewer  patient  recalls  for  repeat 
examinations . 

There  is  another  benefit,  which,  although  difficult  to  quantify,  is 
felt  to  be  important:  many  physicians  at  San  Diego  NRMC  believe  that 
CAPOC  improves  the  overall  accuracy  and  consistency  of  ECG  readings. 

B.  Measurement  of  CAPOC  Impacts  on  Service  Effectiveness 

Table  S-2  summarizes  information  on  effectiveness  of  current  and  projected 
automated  services  for  the  18  CAPOC  sites  in  the  Southern  California  area. 
Figure  S-l  plots  the  change  in  service  effectiveness  for  small  and  large  MTFs. 
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Percent  of  ECGs 
Meeting  Criteria 
for  Effective  Services 
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Because  of  problems  In  interfacing  equipment,  delays  in  changes  to  the 
communications  system,  and  the  need  to  modify  the  computer  software  to 
permit  automated  storage  and  retrieval  of  ECGs  by  Social  Security  Number 
and  Family  Member  Prefix  Code,  CAPOC  was  not  fully  implemented  in  the 
Fall  of  1980  when  evaluation  data  were  collected.  Where  data  were  not 
available,  the  impacts  of  CAPOC  at  full  implementation  were  projected 
on  the  basis  of  knowledge  of  the  manual  services,  observation  of  the 
system's  performance  in  other  sites,  and  the  program  underway  to  maxi¬ 
mize  system  effectiveness  at  all  sites.  More  detailed  information  on 
the  current  level  of  implementation  and  service  provided  by  CAPOC  is 
contained  in  the  last  section  of  this  summary.  Implementation  status 
at  each  site  is  discussed  in  Chapters  3  and  4  of  this  final  report. 

Baseline  data  on  the  volume  of  ECGs,  the  distribution  of  ECGs  by  in¬ 
terpretation,  and  the  type  of  patients  receiving  ECGs,  were  not  available 
for  all  facilities,  and  estimates  by  staff  were  used  to  characterize 
baseline  services.  More  complete  data  were  available  after  automation 
since  workload  information  was  collected  by  the  system.  The  information 
on  service  effectiveness  shown  in  Table  S-2  is  based  upon  our  judgment  of 
the  best  available  data.  However,  since  the  assumptions  made  may  have 
a  large  effect  on  the  magnitude  of  predicted  service  improvements,  a 
range  of  impacts  has  been  calculated  to  examine  the  sensitivity  of  the 
evaluation  results  to  various  assumptions.  These  are  included  in 
Chapter  5  of  this  final  report. 


Accuracy  of  the  Interpretation 


The  accuracy  of  an  ECG  reading  is  difficult  to  measure,  since  an 
ECG  reading  is  truly  an  "interpretation"  and  often  accuracy  cannot  be 
confirmed  by  other  tests.  Therefore,  the  training  of  the  ECG  reader 
was  used  in  the  evaluation  criteria  as  an  indicator  of  reading  quality 
in  the  quantitative  assessment  of  this  service  attribute.  This  analysis 
was  supplemented  by  the  opinions  of  users  regarding  the  relative  accuracy 
of  CAPOC  and  manual  ECG  interpretations. 

Table  S-2  shows  the  quantitative  analysis  of  the  impact  of  CAPOC  on  this 
aspect  of  ECG  reading  in  the  Southern  California  area.  When  ECG  services 


10 


Arthur  D  little  Inc 


in  the  Southern  California  area  were  provided  in  the  manual  mode,  42  percent 
of  ECGs  were  read  by  a  cardiologist  (the  preferred  reader)  and  85  percent 
were  read  by  a  cardiologist  or  internist  (both  effective  readers  accord¬ 
ing  to  the  evaluation  criteria).  Lack  of  access  to  internists  and 
cardiologists  to  read  ECGs  was  a  problem  primarily  at  small  and  remote 
facilities.  With  manual  services  only  32  percent  of  ECGs  in  small  MTFs1 
were  read  by  a  cardiologist  and  only  54  percent  were  read  by  a  cardiolo¬ 
gist  or  internist. 

With  CAPOC,  the  evaluation  criteria  specify  that  the  second  opinion 
provided  by  the  computer  reading  makes  any  physician  or  corpsman  an 
effective  reader  for  a  normal  ECG.  (This  is  consistent  with  the  survey 
results  discussed  later.)  The  computer  reading  support,  plus  the  in¬ 
crease  in  ECGs  transmitted  to  larger  facilities  for  reading,  will  result 
in  an  increase  of  13  percentage  points  (from  85  percent  to  98  percent) 

.lh  the  number  of  ECGs  in  the  18  CAPOC  sites  meeting  the  effective  reader 
criteria  when  automated  services  are  fully  implemented.  The  increase  in 
ECGs  meeting  the  criteria  for  effective  reader  will  be  realized  primarily 
in  small  MTFs. 

Physicians  at  San  Diego  Naval  Regional  Medical  Center  involved  with 
ECG  services  were  asked  to  compare  the  accuracy  of  CAPOC  and  manual  ECG 
reading.  These  physicians  had  the  most  experience  with  CAPOC  since  they 
read  the  largest  number  of  ECGs  and  CAPOC  capability  had  been  available 
at  San  Diego  NRMC  for  a  full  year  when  evaluation  data  were  collected. 

Their  responses  regarding  the  relative  accuracy  of  manual  and  computer- 
assisted  reading  are  summarized  in  Table  S-3.  Of  29  physician  readers, 
who  expressed  an  opinion  about  the  overall  accuracy  of  computer-assisted 
interpretations,  23  (79  percent)  believe  that  CAPOC  ECG  reading  is  at 
least  as  accurate  as  manual  reading.  For  ECGs  within  normal  limits, 

30  of  31  physician  readers  reported  that  CAPOC  was  at  least  as  accurate  as 
manual  reading,  with  13  of  those  indicating  that  CAPOC  was  more  accurate. 

For  abnormal  ECGs,  the  physicians'  opinions  concerning  the  accuracy  of 
CAPOC  and  manual  reading  were  divided,  with  some  favoring  one  or  the 
other  for  different  types  of  abnormalities. 


■'"Facilities  with  fewer  than  100  beds  and/or  fewer  than  200,000  outpatient 
visits  per  year. 


11 


Arthur  1)1  jttle  Inc 


TABLE  S-3 

ACCURACY  OF  CAPOC  COMPARED  WITH  MANUAL  ECG  INTERPRETATIONS— 


0) 


-O'  O  HS  r-^ 

cl  <N  m 


O  -a  ft  -y 

ui  cl  CM  ro 


rn  e*5 

•a-  m 


OS  SO 

c\l  ro 


oo  -a  cm  vo  o  r*. 

os  so  in  so  co  so 


O  so 

m  so 


so  CM 

SO  SO 


r**  r-»  so 

m  so  oo 


0) 

05  V 

d  *H 

o  u 


U  «  C  P  to 

U  W  -H  -H  -H 

•rt  05  d.  U  T3 

05  M  XZ  O  -H 

,fl  (J  o  05  >s 

4-1  04  d  >s  05 


£  C  «  £  £ 
woo  a.  4-i 
o  oo 

iw  W  d  C  r-1  T3 

o  CJ  'H  05  05 

05  *"■»  05  05  AJ 

0  55  O  «  O  O 

u  d  044-10 
W  <0  T3  W  -H 

<0  p.  Os  T3 

4-1  05  .B  X  <N  d 

05  W  05  -H 

W  04  O  T3 

a.  x;  05  a)  w 

W  4-1  3  4-1  T3  O 

05  O  d  3 

4-1  «  JIH  C 

d  >s  3  T3  O  O 

H  55  o  d  d  -H 

■H  O  *H  d 

w  -a  o  a.  -ri 

O  0)  *H  05  d  o. 

N  U  «1  O  O  o 

■a  to  fn  w  *ri 

d  05  xx  *j  o 

p  w  cu  *4h  05  d 

4-1  O  05 
05  O  3  55 
4J  05  f''  M  or  0) 

d  4-1  05  05 

0)  d  TJ  X}  Xt  05 

•H  01  05  d  O  05 

4-1  vH  XI  3  P  5-1 

CO  4-i  3  d  01  p- 

a.  co  >-h  X 

(X  O  0)  W  05 


p-  Os  T3 
X  CH  d 


u  c  x:  o 

-H  W  iH  H  C4H  05 

05  O  4-4 

x:  *«  tu  05  d 

4-1  rW  .  05  05 

o)  a.  o  to  -a 

d  o  §  p  ,n  d 

O  O  CO  CW  O 

w  in  <  u  a 

d  o  d  co 

05  4-4  4-4  0)  0) 

Ai  05  C  <4-1  tJ  W 

co  u  o)  o  d 

4-1  Q,T3  O 

tT  d  d  a.  w 

0)  05  o  O  05  05 

,0  44  a  “rl  4)  d 


05 

cd 

o 

4-J 

rH 

05 

04 

iH 

to 

£  . 

£ 

u 

05 

H 

to 

w 

c 

•  0) 

>s  05 

01  H  01  «  *5 

T3  U  W  JC  XI  d 

W  -r-l  05  4-1  00  (0 

O  05  x:  "H  v-4 

>s  AJ  O  >-l  O 

O  £  O  *5  05  -H 

-do,  oi 

5  >,  W  T) 

05  XI  o  05  ,C 

SW  H  H  a 

CO  05  5-1  W 

co  d  ex  to  in 

•H  05  O  >  I— I 

o  u  d  y 
•H  05  SCO 
05  XI  05  «  °  4J 

X  n  cZ'H 
Xl  <u  o  4-i  <r 

ad  Tt  o  oi  i-i 

4J  M  0) 

05  O  01  3  so 

W  .  4J  *H  C*  d 

p  05  05  Q  P 

4-1  w  _  JC 
05  d  (X  S  05  w 

5-1  05  W  p  P  0)  • 

(I  H  HIM  01  T3  3 

in  44  44  P  O 

P  p  d  u  c  p  d 

It  CX-H  P  O  U  J2 


Arthur  D  Little  Inc 


In  addition,  most  of  the  physician  ECG  readers  at  San  Diego  NRMC 

(24  of  34  who  expressed  an  opinion)  felt  that  the  computer  errors  tended 

to  be  false-positive  errors  (calling  a  normal  ECG  abnormal)  rather  than 

false-negative  errors.  This  is  consistent  with  the  reported  experience 

2 

with  computer-assisted  ECG  reading  elsewhere. 

2.  Turnaround  Time  to  Receive  Interpretations 

At  sites  lacking  a  specialized  ECG  reader,  either  the  patient  or 
the  electrocardiogram  must  sometimes  be  sent  to  another  facility  to 
obtain  an  ECG  interpretation.  In  the  manual  mode,  ECGs  were  most 
commonly  sent  by  surface  or  guard  mail,  read,  and  then  returned.  In  most 
cases  these  ECGs  failed  to  meet  turnaround  time  criteria  for  effective 
services  because  the  ECG  interpretation  was  not  received  back  for  several 
days.  Therefore,  if  an  ECG  from  a  small,  remote  MTF  was  read  by  an 
effective  reader,  it  typically  did  not  meet  criteria  for  effective  turn¬ 
around  time. 

With  manual  services,  11  percent  of  all  ECGs  taken  in  small  facilities 
in  the  Southern  California  area  did  not  meet  the  criteria  for  turnaround 
time.  In  the  Fall  of  1980  only  5  percent  of  ECGs  in  small  facilities  did  not 
meet  turnaround  criteria.  With  full  automation,  this  will  increase  to 
6  percent. 

3.  Use  of  Serial  Comparison  with  Prior  ECGs 

Serial  comparison  refers  to  comparing  a  current  ECG  with  a  prior  one 
in  order  to  identify  changes  that  have  occurred  over  time.  Serial  com¬ 
parisons  are  performed  both  for  hospitalized  patients  (e.g.,  to  monitor 
progress  in  patients  who  have  had  heart  attacks)  and  for  outpatients  to 
check  on  their  cardiac  status.  A  serial  comparison  can  be  a  useful 
screening  tool  since  it  permits  the  physician  to  differentiate  an  actual 
abnormality  from  a  variation  in  an  ECG  that  is  normal  for  a  given  patient 
(a  normal  variant).  In  the  military  health  care  system,  some  annual 


A.  D.  Hagen  and  J.  S.  Alper,  "Evaluation  of  Computer  Programs  for  Clinical 
Electrocardiography."  Circulation  51-52,  Suppl.  11-193,  1975. 
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physical  examinations  also  include  an  electrocardiogram  with  a  serial 
comparison  with  prior  tracings. 

Hospitals  with  a  high  volume  of  electrocardiograms  typically  store 
a  duplicate  copy  of  the  electrocardiogram  in  a  patient  file  in  the  ECG 
department.  When  a  patient  receives  an  ECG,  the  file  of  prior  tracings 
is  retrieved.  In  small  facilities,  the  ECG  copy  in  the  medical  record 
is  often  the  only  source  of  prior  ECGs.  The  ECG  service  criteria  specify 
that  all  abnormal  ECGs  should  receive  a  serial  comparison.  Storage  and 
retrieval  of  a  large  volume  of  ECGs  is  time  consuming  and  often  inaccu¬ 
rate  since  the  patient  may  not  remember  having  prior  ECGs,  the  tracing 
may  be  misfiled,  or  the  medical  record  may  not  be  available.  Therefore, 
automated  storage  and  retrieval  of  ECGs  could  save  technician  time  and 
increase  the  availability  of  prior  tracings . 

The  feature  of  automated  storage  and  retrieval  of  ECGs  is  not  yet  imple¬ 
mented  on  the  CAPOC  system  at  San  Diego  NRMC.  When  it  is  available,  the 
number  of  abnormal  ECGs  in  the  Southern  California  area  that  receive  a  com¬ 
parison  with  a  prior  tracing  is  expected  to  increase  from  43  percent  to 
74  percent.  The  percentage  of  abnormal  ECGs  receiving  a  serial  compari¬ 
son  will  not  reach  100  percent  since  patients  receiving  their  first  ECG 
in  an  MTF  will  have  no  prior  ECG  on  file  for  comparison.  This  of  course 
would  also  be  true  for  a  manual  ECG  process. 

4.  Impact  on  Service  Effectiveness 

The  CAPOC  system  will  have  an  impact  on  effectiveness  of  ECG  services  in 
all  sites  in  the  Southern  California  area;  however,  the  most  dramatic  change 
in  the  level  of  service  effectiveness  will  occur  in  small  facilities 
that  do  not  have  cardiologists  or  internists  on  the  staff  (see  Figure  S-l) . 
At  five  of  18  sites  in  the  Southern  California  area,  none  of  the  baseline 
ECGs  met  criteria  for  effective  services.  At  five  other  sites,  less  than  50 
percent  of  the  ECGs  taken  in  the  baseline  period  met  effective  service 
criteria.  After  automation,  86  percent  of  ECGs  in  small  MTFs  will  meet 
all  criteria  for  effective  service.  Because  a  large  percentage  of  ECGs 
taken  in  these  MTFs  are  normal  and  taken  as  part  of  routine  examinations, 
the  service  requirements  at  these  sites  are  different,  With  CAPOC  providing 
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support  to  the  non-cardiologist  reader  and  the  potential  for  rapid  con¬ 
sultation  with  specialists,  the  effectiveness  of  ECG  services  will  be 
higher  in  small  MTFs  than  in  large  MTFs  when  CAPOC  is  fully  operational. 


The  CAPOC  system  clearly  meets  its  objective  of  extending  high-quality 
ECG  services  to  MTFs  without  a  cardiology  staff. 

C.  Cost-Effectiveness  of  ECG  Services 

To  be  judged  high-quality  (or  "effective")  service,  an  electro¬ 
cardiogram  must  meet  all  service  criteria.  As  shown  in  Table  S-2,  50  per¬ 
cent  of  all  ECGs  in  the  Southern  California  area  met  all  criteria  for  effec¬ 
tive  services  in  the  baseline  period.  With  the  CAPOC  system,  it  is 
predicted  that  78  percent  of  ECGs  will  meet  all  criteria  for  effective 
services.  The  change  in  ECGs  meeting  preferred  service  criteria  will 
be  greater:  7  percent  of  ECGs  met  preferred  criteria  with  manual 
services  and  51  percent  will  meet  preferred  criteria  when  the  CAPOC 
system  is  fully  implemented. 

Hardware  and  communication  costs  associated  with  CAPOC,  though 
partially  offset  by  cost  savings  for  labor  and  supplies,  increased 
the  cost  of  providing  ECG  services  in  the  region  by  21  percent. 

These  costs  are  primarily  for  the  computer  hardware,  analysis  software, 
and  telephone  transmission  of  the  ECGs.  However,  the  percentage  increase 
in  effective  services  is  greater  than  the  percentage  increase  in  the 
cost  of  automation.  Thus,  when  total  costs  of  providing  ECG  services 
are  divided  by  the  number  of  effective  ECGs,  the  cost  per  ECG  meeting 
the  criteria  for  effective  services  will  decrease  from  $15  to  $12  due 
to  the  CAPOC  system.  The  cost  per  ECG  meeting  the  criteria  for  preferred 
ECG  services  will  decrease  from  $107  to  $18.  Therefore,  the  conclusion 
of  this  evaluation  of  the  CAPOC  project  is  that:  when  fully  implemented 
CAPOC  will  provide  more  cost-effective  ECG  services  than  were  provided 
by  manual  methods. 
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D.  Increased  Satisfaction  with  ECG  Services 

Health  care  providers  were  surveyed  at  primary  evaluation 
sites  in  order  to  assess  their  satisfaction  with  ECG  services  before  and 
after  automation.  Physicians  rated  the  following  as  the  most  important 
attributes  to  an  effective  ECG  service: 

•  ability  to  obtain  emergency  ECG  services  24  hours  per  day; 

•  rapid  availability  of  emergency  ECGs; 

•  access  to  ECGs  in  all  treatment  areas; 

•  accuracy,  consistency,  legibility,  and  completeness  of  inter¬ 
pretations; 

•  technical  quality  of  the  tracings ; 

•  availability  of  prior  tracings  for  serial  comparison;  and 

•  availability  of  ECG  expertise  for  consultations. 

In  the  baseline  survey,  physicians  expressed  some  dissatisfaction 
in  each  of  these  areas.  They  were  most  dissatisfied  with  access  to 
emergency  ECG  services  and  the  availability  of  prior  tracings  for 
serial  comparison. 

The  CAPOC  system  is  designed  to  increase  access  to  ECG  interpreta¬ 
tions;  improve  completeness,  consistency,  and  legibility  of  interpre¬ 
tations;  improve  technical  quality  of  tracings;  and  provide  prior  tracings 
for  serial  comparison.  Therefore,  satisfaction  with  ECG  services  is 
expected  to  improve  when  CAPOC  is  fully  operational. 

At  San  Diego  NRMC,  where  physicians  have  had  the  most  experience 
with  the  CAPOC  system  (except  for  the  feature  of  automated  storage  and 
retrieval),  ECG  readers  were  asked  for  their  overall  assessment  of  the 
system.  Of  45  respondents,  47  percent  said  they  were  extremely  or  very 
satisfied,  another  42  percent  were  fairly  satisfied  with  CAPOC  as  cur¬ 
rently  implemented,  and  only  11  percent  were  somewhat  or  very  dissatisfied. 
Of  physicians  who  only  use  ECG  interpretations  in  patient  care  (not  ECG 
readers),  30  percent  were  extremely  or  very  satisfied,  and  55  percent 
were  fairly  satisfied.  Only  15  percent  were  somewhat  unsatisfied,  and 
none  reported  being  very  dissatisfied  with  CAPOC.  Reasons  for  dissatis¬ 
faction  were  related  to  technical  problems  (outdated  equipment)  or 
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A.  Current  Status 


Several  factors  have  delayed  full  implementation  of  CAPOC  in  the  Southern 
California  area.  The  capability  for  automated  storage  and  retrieval  of 
ECGs  needed  to  be  adapted  to  the  11-digit  patient  identification  system 
used  in  the  military.  Therefore,  this  feature  of  the  system,  which  will 
facilitate  use  of  serial  comparisons,  was  not  in  use  during  Y  Period 
data  collection.  Availability  of  automatic  storage  and  retrieval  will 
increase  the  effectiveness  of  services  in  12  of  the  18  sites  in  the 
region. 

Technical  problems  in  transmission  to  the  central  computer  have 
been  encountered  in  five  sites.  This  has  been  attributed  to  the  inter¬ 
facing  of  some  ECG  carts  with  the  CAPOC  system  and  to  communication 
problems.  Many  of  the  sites  that  have  experienced  continual  transmission 
difficulties  have  delayed  use  of  CAPOC  until  the  problems  can  be  resolved. 

At  other  sites  use  of  CAPOC  is  intermittent. 

ECG  capture  at  three  sites  has  been  limited  because  CAPOC  capability 
is  not  available  in  all  areas  where  ECGs  are  taken.  One  other  site  has 
had  maintenance  difficulties  with  an  old  ECG  machine.  A  total  of  five 
more  ECG  machines  in  the  region  would  increase  the  percentage  of  ECGs 
affected  by  CAPOC. 

Some  change  in  department  operations  would  also  maximize  the  bene¬ 
fits  of  CAPOC.  If  unconfirmed  computer  reports  for  normal  ECGs  were  re¬ 
turned  with  the  ambulatory  patients  and  abnormal  ECGs  for  pre-operative 
patients  were  available  as  soon  as  they  were  read,  the  percentage  of 
ECGs  meeting  criteria  for  effective  turnaround  time  would  increase. 

Programs  are  currently  underway  to  solve  implementation  problems  in  the 
Southern  California  area  in  order  to  obtain  maximum  benefits  from  the 
system. 
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B.  Achieving  Full  Implementation  of  CAPOC 

During  this  evaluation,  technical  and  operational  changes  were 
identified  that  would  increase  the  benefits  of  the  CAPOC  system  in  the 

Southern  California  area: 

•  technical  enhancements; 

•  better  equipment  availability  and  reliability; 

•  additional  training  in  system  use;  and 

9  changes  in  operating  procedures. 

The  major  technical  enhancement  that  will  increase  the  system's  cost 
effectiveness  involves  hardware  and/or  software  modifications  to  provide 
capability  for  automatic  storage  and  retrieval  of  ECGs.  Two  modifications 
are  required:  (a)  the  inclusion  of  patient  name  associated  with 

Family  Member  Prefix  and  Social  Security  Number,  and  (b)  automatic  re¬ 
trieval  of  the  most  recent  prior  ECG  tracing  for  purposes  of  serial  com¬ 
parison.  These  modifications  are  currently  in  progress  and  are  expected 
to  be  implemented  by  September  1,  1981.  The  availability  of  automatic 
storage  and  retrieval  is  expected  to  result  in  an  increase  in  the  number 
of  ECGs  that  receive  a  serial  comparison  since,  with  manual  filing,  ECGs 
are  apt  to  be  misfiled  or  lost. 

At  several  sites,  ECG  capture  by  CAPOC  was  limited  because  ECG  carts 
were  not  available  for  use  in  all  locations  where  large  volumes  of  ECGs 
are  taken,  available  equipment  was  outdated,  or  some  equipment  was  not 
functioning  reliably.  At  the  time  of  system  installation,  some  sites 
were  experiencing  problems  with  the  interface  of  existing  carts  with  the 
automated  system  due  to  both  technical  problems  with  the  interface  and 
communications  problems. 

By  July  1981,  problems  with  equipment  interface,  telephone  access, 
and  telephone  transmission  had  been  resolved.  Approximately  four  addi¬ 
tional  carts  are  currently  on  order.  Since  cart  acquisition  is  the 
responsibility  of  the  local  commands,  subject  to  the  availability  of 
procurement  funds,  acquisition  of  all  carts  required  to  maximize  ECG 
capture  by  CAPOC  will  probably  not  be  achieved  until  September,  1981. 
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The  issue  of  cart  maintenance  has  also  been  addressed.  In  a  recent 
visit,  TRIMIS  personnel  noted  a  definite  improvement  in  the  areas  of  cart 
maintenance,  training,  and  availability.  The  knowledge  of  site  staff  of 
equipment  maintenance  is  considered  to  be  very  good.  In  addition,  TPO 
has  recently  established  a  formal  dialogue  with  senior  Marquette  and 
Hewlett-Packard  personnel  and  received  assurances  from  both  firms  of  an 
on-going  commitment  to  provide  fast  and  effective  service,  where  required. 
A  procedure  is  being  established  that  will  appraise  TPO  of  cart  main¬ 
tenance  problems  so  that  appropriate  action  may  be  taken  as  quickly  as 
possible. 

The  Regional  CAPOC  System  Manager,  located  at  San  Diego  NRMC,  has 
instituted  an  aggressive  training  and  information-sharing  program  on  a 
region-wide  basis.  The  program  is  extremely  effective  and  is  meeting 
with  wide  acceptance  by  all  users.  The  Regional  Manager  has  initiated 
the  publication  of  a  monthly  "CAPOC  Newsletter,"  which  provides  a  forum 
for  the  interchange  of  ideas  and  problem  resolution,  as  well  as  a 
vehicle  for  informing  (and  reminding)  users  about  matters  such  as  system 
capabilities  and  projected  changes.  The  Regional  Manager  has  also  in¬ 
stituted  a  training  program  so  personnel  from  remote  sites  may  travel  to 
San  Diego  NRMC  for  refresher  and/or  familiarization  training.  When 
necessary,  the  Regional  Manager  can  also  provide  training  at  the  user 
site.  To  date,  the  Manager  has  made  five  training  visits  and  this 
appears  to  be  an  effective  tool  for  maintaining  a  desirable  level  of 
user  understanding.  With  facilities  available  at  San  Diego  NRMC,  a 
video  cassette  has  been  produced  describing  system  procedures  and  capa¬ 
bilities.  This  film,  and  additional  cassettes  concerning  ECG  cart  input 
and  CAPOC  editing,  will  be  used  as  an  aide  for  training  of  newly  assigned 
personnel  and  in  conducting  refresher  courses  for  current  personnel. 

Operational  procedures  have  been  modified  at  some  of  the  sites  so 
that  the  unconfirmed  CAPOC  interpretations  of  normal  ECGs  are  returned 
immediately  to  the  physician  who  ordered  the  ECG.  Other  operational 
changes  required  to  enhance  CAPOC  benefits  are  being  undertaken. 
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The  actions  underway  to  maximize  the  benefits  of  CAPOC  in  the  18 
MTFs  in  the  Southern  California  area  have  been  extremely  successful  to 
date.  Resolution  of  equipment  interface,  cart  availability,  and 
training  problems  has  enabled  six  sites  to  begin  transmitting  ECGs  to 
the  system.  This  will  greatly  enhance  ECG  capture  by  CAPOC  in  the 
region.  The  active  involvement  of  the  CAPOC  site  staff,  the  Regional 
Manager,  TRIMIS  representatives  from  each  service,  and  the  TPO  will 
result  in  the  further  changes  that  are  necessary  to  realize  maximum 
benefits  from  the  system. 
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CHAPTER  1.  INTRODUCTION 


A.  OVERVIEW 


The  Tri-Service  Medical  Information  Systems  (TRIMIS)  Program  Office 
(TPO)  has  procured  a  computerized  electrocardiography  (ECG)  system  as 
part  of  a  program  known  as  CAPOC  (Computer-Assisted  Practice  of  Cardio¬ 
logy)  .  The  overall  objective  of  the  CAPOC  project  is  "to  improve  cardiac 
diagnosis  and  treatment  in  the  military  health  care  system  by  extending 
high  quality  ECG  analysis  service  to  a  larger  number  of  MTFs,  many  of 
which  do  not  have  cardiology  staffs."  Specific  objectives  for  the 
program  outlined  by  the  TRIMIS  Medical  Review  Group  include: 3 

•  "To  make  enhanced  ECG  interpretation  available  to 
cardiologist  and  non-cardiologist  requesters  with 
increased  efficiency  and  accuracy  in  the  detection 
and  treatment  of  cardiac  disease; 

•  To  have  the  capability  for  handling  increased  demand 
for  ECG  interpretation  without  a  significant  increase 
in  staff; 

•  To  reduce  the  clerical  work  required  of  qualified  ECG 
technicians  in  the  cardiology  service; 

•  To  gather  as  a  result  of  normal  operations,  workload 
and  managerial  data  and  to  present  these  as  required 
in  order  to  assist  in  resource  decisionmaking  in 

the  MTF; 

•  To  provide  accountability  for  ECG  requests  and  to 
monitor  the  generation  of  interpretation  results,  to 
include  providing  notices  of  abnormal  ECG  findings  or 
improper  quality  control  as  soon  as  they  are  detected; 
and 

•  To  reduce  the  number  of  false  negatives  (reading  an 
abnormal  ECG  as  normal)  interpreted  by  non-cardiologist 
physicians. 


•^Initial  Project  Objectives  and  Evaluation  Criteria,  TRIMIS  Medical 
Review  Group,  September  i,  iy//,  pages  23=277 


♦ibid 
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The  equipment  for  the  CAPOC  system  was  installed  by  the  vendor, 
Marquette  Electronics,  in  the  18  Medical  Treatment  Facilities  (MTFs) 
in  the  Southern  California  area  beginning  in  October  1979.  If  the 
initial  CAPOC  installations  are  successful,  TRIMIS  plans  eventually  to 
install  14  more  regional  computer  systems  serving  MTFs  throughout  the 
United  States. 

B.  EVALUATION  OBJECTIVES  AND  METHODOLOGY 

Arthur  D.  Little,  Inc.,  has  evaluated  the  installation  of  CAPOC 
in  the  Southern  California  area  in  order: 

•  to  provide  TPO  with  information  needed  for  the  pro¬ 
liferation  decision; 

•  to  determine  whether  the  system  met  the  objective 
of  extending  high  quality  ECG  analysis  services  to 
MTFs  without  cardiology  staffs; 

•  to  determine  if  automated  services  are  cost-effective;  and 

•  to  identify  changes  that  should  be  made  to  increase 
the  effectiveness  of  the  system  or  to  lower  its 
cost. 

This  assessment  has  been  accomplished  by  a  comparison  of  the 
costs,  benefits  and  acceptance  of  ECG  services  before  and  after  the 
installation  of  CAPOC  in  the  18  MTFs  involved  in  the  initial  system 
installation. 

TRIMIS  representatives  and  evaluation  project  staff  recognized 
that  in  addition  to  improving  ECG  services,  CAPOC  would  increase  the 
total  cost  of  ECG  services.  Many  of  the  important  anticipated  benefits 
would  likely  be  improvements  in  the  quality  of  ECG  services  provided. 

In  order  to  relate  quantitatively  changes  in  ECG  services  with 
changes  in  costs,  an  approach  was  developed  that  would  permit  a  mean¬ 
ingful  comparison  of  cost-effectiveness  of  services  before  and  after 
CAPOC. 
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The  approach  developed  involved  measuring  the  number  of  ECGs  at 
each  site  that  met  pre-established  service  criteria  in  order  to  quantify 
the  level  (effectiveness)  of  service  being  delivered.  A  panel  of  mili¬ 
tary  physicians  helped  to  develop  the  criteria  for  effective  ECG  services 
to  be  used  in  the  evaluation.  These  criteria  established  effective 
service  levels  for  type  of  staff  interpreting  ECGs,  turnaround  time,  and 
use  of  serial  comparison  of  prior  ECGs  with  current  ones.  The  evaluation 
effort  has  been  focused  upon  the  collection  of  data  that  would  allow  a 
determination  of  the  cost  per  ECG  meeting  effective  service  criteria 
before  and  after  CAPOC  implementation. 

In  addition  to  the  quantitative  assessment  of  the  relative  costs 
and  performance  of  manual  and  CAPOC  ECG  services,  system  impacts  that 
are  not  readily  quantified  were  evaluated  by  use  of  interviews  and  sur¬ 
veys.  These  factors  included  staff  acceptance  of  CAPOC,  staff  percep¬ 
tions  of  strengths  and  weaknesses  of  manual  vs.  automated  services,  the 
relative  ingjortance  of  ECG  service  attributes,  and  patient  referral  and 
recall  patterns. 

CAPOC  implementation  was  planned  in  a  total  of  18  MTFs .  In  order 
to  conduct  the  evaluation  at  a  reasonable  level  of  effort,  five  of  the 
MTFs  were  selected  as  primary  evaluation  sites  and  studied  in  great 
detail: 

(1)  San  Diego  Naval  Regional  Medical  Center  (San  Diego  NRMC) , 

(2)  Camp  Pendleton  Naval  Regional  Medical  Center  (Camp  Pendleton), 

(3)  USAF  Hospital,  Nellis  Air  Force  Base  (Nellis), 

(A)  Twentynine  Palms  Marine  Corps  Base  Branch  Hospital  (Twenty- 
nine  Palms) ,  and 

(5)  Miramar  Naval  Air  Station  Branch  Clinic  (Miramar) . 

These  sites  represent  the  range  of  staffing  patterns,  patient  popu¬ 
lations,  and  ECG  services  in  the  region. 
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So  that  conclusions  could  be  reached  about  the  system's  costs  and 
benefits  in  the  entire  region,  basic  information  on  ECG  services  was 
also  collected  from  each  of  the  remaining  13  medical  treatment  facilities 
in  the  region  that  were  included  in  the  CAPOC  system  installation.  These 
secondary  evaluation  sites  include: 

(1)  USAF  Hospital,  Vandenberg  Air  Force  Base  (Vandenberg) , 

(2)  USAF  Hospital,  Edwards  Air  Force  Base  (Edwards), 

(3)  USAF  Hospital,  George  Air  Force  Base  (George), 

(4)  USAF  Medical  Facility,  Norton  Air  Force  Base  (Norton) 

(5)  USAF  Regional  Hospital,  March  Air  Force  Base  (March), 

(6)  Long  Beach  Naval  Regional  Medical  Center  (Long  Beach), 

(7)  Naval  Station  San  Diego  Branch  Clinic  (Naval  Station), 

(8)  Naval  Air  Station  North  Island  Branch  Clinic  (North  Island) , 

(9)  Naval  Training  Center  Branch  Clinic  (NTC) , 

(10)  Naval  Amphibious  Base  Coronado  Branch  Clinic  (Coronado) , 

(11)  Marine  Corps  Recruiting  Depot  Branch  Clinic  (MCRD) , 

(12)  Port  Hueneme  Naval  Regional  Medical  Clinic  (Port  Hueneme) ,  and 

(13)  Barstow  Branch  Clinic  (Barstow) . 

Information  was  obtained  at  these  sites  from  available  records  and 
staff  interviews  rather  than  by  extensive  collection  of  original  data. 

ECG  service  characteristics  at  these  sites  were  matched  with  those  at 
the  five  primary  sites  so  that  changes  due  to  CAPOC  could  be  estimated 
based  upon  similarities  in  ECG  service  characteristics. 

Finally  changes  in  costs  and  effectiveness  were  summed  for  all  18 
MTFs  in  order  to  perform  a  regional  analysis  of  the  costs  and  effective¬ 
ness  of  ECG  services  before  and  after  the  installation  of  CAPOC. 

C.  DESCRIPTION  OF  ECG  TECHNOLOGIES 

For  those  readers  unfamiliar  with  computerized  ECG  technologies,  the 
following  brief  description  is  included  to  provide  the  background 
necessary  for  understanding  the  CAPOC  evaluation. 

An  ECG  is  a  recording  of  the  electrical  activity  generated  during 
the  heart's  contraction.  The  ECG  is  obtained  by  attaching  electrodes  to 
the  patient's  arm,  legs,  and  six  positions  across  the  chest.  Since  the 
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body  conducts  electricity,  these  body  surface  electrodes  record  elec¬ 
trical  potentials  originating  within  the  heart  that  are  transmitted  to 
the  surface.  An  ECG  machine  records  these  potentials  by  selecting 
different  combinations  of  electrodes,  known  as  "leads."  The  standard 
ECG  includes  12  leads.  These  leads  can  be  recorded,  one  at  a  time,  on 
a  single-channel  machine,  which  produces  a  long,  narrow  strip  of  paper, 
or  they  can  be  recorded  three  at  a  time  using  a  three-channel  machine, 
which  produces  a  convenient  record  for  filing  (8  1/2  inches  by  11  inches 
in  size). 

In  an  ECG  department  using  a  manual  system,  the  physician  examines 
the  hard-copy  record  generated  by  the  ECG  machine  and  interprets  the 
wave-forms  by  comparing  measurements  of  amplitudes  and  intervals  against 
normal  values  and/or  by  recognizing  patterns  known  to  be  associated 
with  various  cardiac  conditions.  The  reader  is  concerned  with  identify¬ 
ing  patterns  associated  with  cardiac  abnormalities  (heart  attacks,  heart 
enlargement)  and  disturbances  in  heart  rhythm,  known  as  arrhythmias. 

The  physician's  interpretation  is  then  either  written  or  dictated 
for  transcription.  The  interpretation  is  matched  with  the  ECG  tracing 
and  filed  in  the  patient's  medical  record.  Often  a  duplicate  tracing 
and  interpretation  are  kept  in  an  ECG  department  file.  Frequently,  the 
physician  compares  prior  tracings  with  current  ECGs  when  making  an  inter¬ 
pretation  so  that  recent  abnormalities  can  be  identified.  This  is  re¬ 
ferred  to  as  a  serial  comparison. 

Computerized  ECG  technology,  and  the  CAPOC  system  in  particular, 
differs  in  a  number  of  ways  from  the  manual  system.  It  uses  a  three- 
channel  ECG  machine  that  is  equipped  to  transmit  the  ECG  over  telephone 
lines  to  a  computer.  The  computer  processes  these  signals,  makes  measure¬ 
ments  on  the  waveform,  compares  the  measurements  with  criteria  for 
different  conditions ,  and  generates  an  interpretation.  The  interpretation 
(known  as  the  unconfirmed  interpretation)  is  then  transmitted  back  within 
minutes  to  the  user  site,  or  to  any  other  designated  facility.  The  ECG, 
along  with  relevant  patient  information,  is  stored  in  the  computer  (or 
on  a  peripheral  storage  disk)  and  can  be  retrieved  at  a  later  time 
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for  comparison  with  more  recent  ECG  tracings.  The  unconfirmed  interpre¬ 
tation  and  a  hard-copy  record  of  the  ECG  tracing  are  usually  overread  by 
a  physician,  who  either  confirms  or  edits  the  computer  interpretation. 
These  editing  changes  can  be  entered  into  the  computer  if  user  sites 
are  equipped  with  a  data  entry  terminal. 

Use  of  electrocardiograms  in  the  military  is  in  most  ways  quite 
similar  to  the  use  of  ECG's  in  civilian  medical  practice.  There  are  a 
few  differences  related  to  the  needs  of  the  military  population  and  the 
resources  available  to  deliver  care.  Since  ECGs  are  a  non-invasive 
and  relatively  inexpensive  means  of  evaluating  cardiac  status,  they 
are  part  of  the  basic  cardiac  examination  for  many  different  types  of 
patients.  Electrocardiograms  are  used  to  evaluate  emergency  patients 
with  suspected  myocardial  infarctions  (heart  attacks);  they  are  also 
used  to  identify  cardiac  problems  such  as  prior  myocardial  infarctions 
or  abnormal  heart  rhythms.  The  ECG  is  also  used  in  evaluating  apparently 
healthy  individuals  to  rule  out  the  possibility  of  any  unsuspected 
problems.  For  instance  pre-operative  ECGs  are  taken  to  determine  whether 
there  are  unsuspected  cardiac  problems  that  should  be  known  during 
surgery.  ECGs  are  also  routinely  taken  as  part  of  a  routine  physical 
examinations,  especially  in  older  male  patients.  In  the  military, 
routine  ECGs  are  also  included  as  part  of  some  physical  examinations 
required  for  the  active-duty  population. 

Because  of  the  many  different  types  of  patients  who  require  ECGs, 
and  because  electrocardiograms  are  taken  in  emergency  situations,  it 
is  important  to  have  access  to  electrocardiography  services  in  all 
military  treatment  facilities.  The  military  does  not  have  enough 
cardiologists  (physicians  who  are  expert  in  reading  electrocardiograms) 
to  staff  every  military  treatment  facility.  Many  military  facilities 
are  also  in  remote  areas  where  there  are  no  civilian  cardiologists 
available  for  back-up.  The  CAPOC  system,  which  produces  a  communication 
network  giving  small  facilities  access  to  consultation  from  cardiologists 
in  large  facilities,  is  seen  as  a  way  to  provide  high-quality  ECG  services 
even  for  emergency  patients  in  facilities  without  cardiologists  on 
their  staff. 
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As  will  be  explained  later  in  this  report  text,  the  automated  ECG 
services  also  provide  additional  benefit  in  helping  to  manage  the  large 
ECG  departments,  improve  storage  and  access,  save  technician  time,  and 
improve  the  quality  of  the  electrocardiogram  tracing.  However,  the 
primary  objective  of  the  CAPOC  system  is  to  provide  service  to  small, 
remote  facilities  who  use  electrocardiograms  in  patient  care,  but  do 
not  have  ready  access  to  a  cardiologist  for  interpreting  the  tracings. 

D.  OVERVIEW  OF  EVALUATION  ACTIVITIES 

The  CAPOC  evaluation  was  begun  by  Arthur  D.  Little,  Inc.  in  June 
1978.  CAPOC  goals  and  objectives  were  reviewed  and  initial  site  visits 
were  conducted  to  the  five  primary  evaluation  sites.  These  visits  showed 
where  data  could  be  obtained  and  provided  the  necessary  background  for 
understanding  where  impacts  due  to  CAPOC  would  be  likely  to  occur.  An 
initial  evaluation  plan  was  prepared  in  October  1978.-*  From  this  plan 
evolved  the  final  approach  for  determining  the  cost-effectiveness  of 
CAPOC  in  contrast  to  the  manual  system,  which  it  has  replaced. 

The  evaluation  of  ECG  services  involves  several  elements:  measur¬ 
ing  the  utilization  of  services,  characterizing  the  level  of  services 
provided,  documenting  costs  of  services,  and  surveying  staff  about  satis¬ 
faction  with  services  and  areas  that  could  be  improved. 

Both  quantitative  and  qualitative  data  were  needed  in  order  to  con¬ 
duct  a  complete  evaluation.  For  each  of  the  five  primary  evaluation 
sites,  a  specific  plan  for  collecting  quantitative  data  was  developed. 

One  set  of  questionnaires  was  prepared  for  obtaining  qualitative  data 
from  all  five  sites.  The  methods  for  quantitative  and  qualitative  data 
collection  were  pre-tested  and  revised.  For  the  13  secondary  evaluation 
sites,  one  interview  guide  was  prepared  and  used  for  collecting  con¬ 
sistent  information  from  all  sites. 


^"Research  Plan  for  Measuring  the  Impact  of  CAPOC  in  the  San  Diego 
Region."  Prepared  for  the  TRIMIS  Program  Office  by  Arthur  D.  Little, 
Inc. ,  October  1978.  Prepared  under  a  subcontract  with  the  Johns 
Hopkins  University  Applied  Physics  Laboratory,  Contract  No.  APL600869. 
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Between  August  and  October  1979,  "X"  Period  data  were  collected 
at  the  five  primary  sites.  In  April  and  May  1980,  additional  data  were 
collected  at  these  five  sites,  and  interviews  were  conducted  at  the 
other  13  sites. 

Installation  of  the  CAPOC  system  began  in  October  1979.  In  Septem¬ 
ber  and  October  1980,  "Y"  Period  data  were  collected,  reflecting  the 
status  of  CAPOC  implementation  1  year  after  system  installation  was 
begun.  Extensive  data  and  information  were  collected  as  available  at 
the  five  primary  evaluation  sites,  and  each  of  the  13  secondary  evalua¬ 
tion  sites  was  visited  in  order  to  assess  the  status  of  system  implemen¬ 
tation  and  to  characterize  the  CAPOC  ECG  process.  These  data  were 
analyzed. 

This  report  presents  the  data  analysis  and  findings  of  the  evaluation: 

(1)  by  comparing  the  costs  and  effectiveness  of  the  manual  and 
CAPOC  ECG  processes  in  the  Southern  California  area,  and 

(2)  by  assessing  other  qualitative  impacts  of  CAPOC  as  per¬ 
ceived  by  system  users. 

Because  of  problems  with  equipment  interfaces,  the  automated  storage 
and  retrieval  function,  and  operational  changes  in  the  ECG  process 
needed  at  some  sites,  CAPOC  was  not  fully  implemented  in  the  Fall  of 
1980  when  evaluation  data  were  collected.  Where  data  were  not  available, 
the  impacts  of  CAPOC  at  full  implementation  were  projected  on  the  basis 
of  knowledge  of  the  manual  services,  observations  of  the  system’s 
performance  in  other  sites,  and  plans  underway  to  achieve  full  imple¬ 
mentation  at  all  sites. 

ORGANIZATION  OF  THE  REPORT 


The  remainder  of  this  evaluation  report  is  comprised  of  five  chapters: 

•  Chapter  2  discusses  the  methods  of  data  collection  and  analysis, 

•  Chapter  3  presents  data  and  information  collected  related  to 
ECG  volumes  and  service  effectiveness  of  the  five  primary 
evaluation  sites, 

•  Chapter  4  summarizes  information  regarding  CAPOC  implementation 
at  the  13  secondary  evaluation  sites,  and 

•  Chapter  5  presents  results  regarding  service  effectiveness, 
costs,  and  cost  effectiveness. 
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CHAPTER  2 .  METHODOLOGY 


A.  OVERALL  APPROACH 


The  overall  approach  taken  in  the  CAPOC  evaluation  was  a  before  and 
after  study  to  document  costs,  benefits,  and  staff  acceptance  of  the  sys¬ 
tem.  The  major  features  of  the  methodology  employed  to  compare  ECG 
services  before  (known  as  the  "X  Period")  and  after  (known  as  the  "Y 
Period")  CAPOC  implementation  were  dictated  by  the  objectives  for  the 
evaluation,  the  anticipated  benefits  of  automating  ECG  services,  and  the 
need  to  conduct  the  evaluation  at  a  reasonable  cost. 

Since  it  was  recognized  that  CAPOC  was  likely  to  increase  the  costs 
of  providing  ECG  services  in  MTFs  in  the  Southern  California  area,  while 
at  the  same  time  improving  the  quality  of  services  provided,  the  evalua¬ 
tion  had  to  value  costs  and  benefits  of  the  system.  Though  changes  in 
ECG  services — such  as  improved  service  response  times,  increased  accuracy 
of  ECG  reading,  and  increased  availability  of  prior  ECGs  for  serial  com¬ 
parison — could  be  measured,  they  were  difficult  to  translate  into  dollar 
value  for  a  direct  comparison  with  system  costs.  For  example,  a  reduction 
in  turnaround  time  for  receipt  of  an  ECG  interpretation  for  a  pre-opera¬ 
tive  patient  from  20  hours  to  4  hours  might  be  of  real  benefit  to  patient 
care,  but  the  value  in  dollars  would  be  difficult  to  estimate.  Similarly, 
an  increase  in  availability  of  prior  ECGs  would  facilitate  use  of  serial 
comparisons,  but  the  value  of  serial  comparisons  to  clinical  decisionmaking 
could  not  be  expressed  easily  as  a  dollar  value.  Therefore,  an  evaluation 
measure  was  needed  that  could  both  capture  significant  changes  in  the  im¬ 
portant  service  benefits  and  relate  them  to  changes  in  costs  in  a  combined 
measure  of  the  usefulness  or  value  of  the  system. 

Discussions  with  users,  combined  with  experience  in  civilian  health 
care  facilities,  indicated  that  CAPOC  would  improve  service  in  three 
major  areas: 

(1)  The  time  required  to  receive  an  ECG  interpretation  would 
decrease  in  remote  sites  (many  of  which  previously  sent 
ECGs  through  the  mail  for  reading) ; 
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(2)  Interpretations  of  ECG  tracings  would  be  more  accurate 
and  consistent;  and 

(3)  Access  to  prior  ECG  tracings  would  improve,  and  the  fre¬ 
quency  with  which  current  ECGs  were  compared  with  prior 
ECGs  in  order  to  identify  changes  would  increase. 

Based  upon  these  service  attributes,  a  panel  of  users  identified  two 
levels  of  effective  ECG  services:  what  is  typically  required  for  the  ECG 
to  be  useful  in  clinical  management  of  patients  (acceptable  ECG  services) 
and  what  permits  the  maximum  benefit  from  ECG  information  in  clinical 
decisionmaking  (preferred  service).  Together  these  two  levels  of  effective 
service  provided  the  evaluation  measure  used  to  quantify  changes  in 
service  quality. 

When  the  measured  changes  in  service  quality  were  combined  with  in¬ 
formation  regarding  the  costs  of  providing  ECG  services  with  and  without 
CAPOC,  the  cost-effectiveness  of  the  two  service  modes  could  be  compared. 

If  the  cost  per  effective  ECG  was  lower  with  CAPOC,  the  system  would  be 
considered  to  have  met  its  objectives. 

The  initial  CAPOC  installation  involved  a  total  of  18  MTFs ,  many 
of  which  provided  ECG  services  in  several  different  locations.  Document¬ 
ing  multiple  system  impacts  in  each  of  these  ECG  service  settings  would 
have  required  a  massive  evaluation  effort.  Therefore,  a  decision  was 
made  to  focus  intensive  evaluation  efforts  on  a  sample  of  carefully 
selected  sites  and  to  project  the  likely  impacts  of  CAPOC  for  the  remain¬ 
der  of  the  sites. 

Primary  evaluation  sites  were  chosen  to  be  representative  of  the 
type  of  staff,  patient  population,  and  ECG  services  in  the  region.  The 
five  selected  included  the  largest  referral  hospital  in  the  region;  one 
smaller  referral  hospital;  one  small  branch  clinic;  one  small,  remote 
branch  hospital;  and  one  small  hospital  that  was  not  affiliated  with  a 
large  center.  The  five  sites  included  Nr'ry,  Marine,  and  Air  Force  facili¬ 
ties  (no  Army  MTFs  are  located  in  the  Southern  California  area). 

Information  on  ECG  services  in  the  primary  evaluation  sites  was  ob¬ 
tained  by  on-site  data  collection,  review  of  hospital  statistics,  inter¬ 
views  with  staff,  and  administration  of  a  staff  survey.  The  additional 
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thirteen  sites  in  the  region  were  visited  briefly,  and  a  description  of 
the  ECG  process  was  obtained.  On  the  basis  of  this  information  and  data 
from  the  five  primary  sites,  levels  and  costs  of  ECG  services  as  provided 
before  and  after  CAPOC  were  estimated  for  the  entire  region. 

In  addition  to  measured  changes  in  service  effectiveness  (by  use  of 
the  set  of  service  criteria) ,  CAPOC  was  expected  to  have  other  important 
benefits  that  are  more  difficult  to  quantify.  Furthermore,  one  objective 
of  the  evaluation  was  to  provide  insight  into  staff  acceptance  of  CAPOC 
and  to  identify  changes  in  the  CAPOC  system  and  its  use  that  would  in¬ 
crease  benefits  and/or  reduce  costs.  Therefore,  the  quantitative  evalua¬ 
tion  was  supplemented  by  a  qualitative  assessment  of  a  number  of  other 
aspects  of  ECG  services.  This  was  accomplished  by  a  survey  of  users  and 
providers  of  ECG  services  at  the  primary  evaluation  sites  and  by  inter¬ 
views  with  staff  at  the  secondary  evaluation  sites. 

In  summary,  ECG  services  before  and  after  CAPOC  installation  were 
compared  by  a  quantitative  and  qualitative  assessment  in  a  sample  of  sites, 
with  the  results  projected  to  the  entire  region.  The  quantitative  evalua¬ 
tion  was  focused  on  the  relative  costs  and  performance  of  manual  and  CAPOC 
services,  as  measured  against  criteria  for  effective  ECG  services. 

The  remainder  of  this  chapter  describes  the  development  of  the  evalua¬ 
tion  criteria  and  the  techniques  for  data  collection  and  analysis. 

B.  DEVELOPMENT  OF  CRITERIA  FOR  EFFECTIVE  ECG  SERVICES 
1.  Identification  of  Important  Service  Parameters 

Establishing  criteria  for  effective  ECG  services  required  identifying 
parameters  that  are  important  in  determining  the  clinical  usefulness  of 
the  ECG.  These  parameters  were  identified  by  reviewing  information  col¬ 
lected  during  the  original  site  visits  and  noting  where  changes  were 
anticipated  and  by  interviewing  staff  at  the  sites  concerning  their  expec¬ 
tations  of  changes  that  would  result  in  benefits  to  patient  care.  Five 
factors  were  identified  by  this  process: 

•  Turnaround  time  (elapsed  time  between  taking  of  the  ECG  tracing 
and  the  return  of  the  interpretation), 

•  Use  of  serial  comparison  (comparing  the  current  ECG  tracing 
with  a  prior  one  and  noting  differences). 
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•  Patient  population  receiving  ECGs, 

•  Type  of  staff  interpreting  the  ECGs,  and 

•  Accuracy,  consistency,  and  completeness  of  the  interpretation. 

The  first  four  factors  were  considered  further  in  this  study.  A 

separate  accuracy  evaluation  is  also  planned. 

2.  Need  for  Service  Levels 

Before  improvements  in  these  parameters  could  be  quantified,  however, 
it  was  necessary  to  distinguish  those  changes  that  would  affect  patient 
care  from  less  significant  changes.  The  need  for  this  differentiation 
is  illustrated  in  the  following  example  regarding  changes  in  service 
response  time. 

A  decrease  in  turnaround  time  from  14  days  to  7  days  or  from  8  hours 
to  4  hours  might  not  result  in  a  100%  increase  in  the  usefulness  of  the 
ECG.  In  order  to  value  this  potential  benefit,  it  is  necessary  to 
examine  how  these  reduced  turnaround  times  might  actually  affect  patient 
care. 

In  the  case  of  an  outpatient  ECG,  three  different  uses  of  the  ECG 
have  been  defined.  First,  if  the  ECG  interpretation  were  returned  very 
quickly  (within  30  minutes),  the  patient  could  wait  at  the  MTF  while  the 
results  were  reviewed.  In  this  case,  the  information  derived  from  the 
ECG  could  be  reflected  immediately  in  the  decisionmaking  of  the  attending 
physician.  If  the  interpretation  were  received  within  3  days,  the 
physician  would  most  likely  still  have  easy  access  to  the  patient's 
chart  and  could  integrate  the  ECG  information  with  other  laboratory  in¬ 
formation,  and  make  a  decision  about  whether  the  patient  should  be  re¬ 
called  for  follow-up  evaluation  or  treatment  initiation.  If,  however, 
the  ECG  were  returned  after  3  days,  the  physician  might  have  to  recall 
the  patient's  chart  from  the  record  room  in  order  to  consider  the  ECG 
information,  in  conjunction  with  patient  history  and  other  test  results, 
or  the  patient  information  might  not  be  reviewed  until  the  next  patient 
visit. 

On  the  basis  of  these  described  differences  in  patient  care,  th.ee 
levels  of  service  could  be  defined  for  the  turnaround  time  for  receipt 
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of  the  interpretation:  within  30  minutes,  within  3  days,  and  over  3 
days.  For  this  theoretical  example,  a  change  in  turnaround  time  from 
1  day  to  30  minutes  would  represent  a  significant  change  since  it  would 
affect  the  way  the  ECG  was  used  in  patient  care.  However,  a  change  from 
8  hours  to  4  hours  would  not  be  considered  significant  since  the  way 
the  ECG  could  be  used  in  patient  care  would  not  be  affected. 

As  this  example  illustrates,  the  information  derived  from  an 
ECG  can  have  various  degrees  of  usefulness  in  patient  care  depending 
upon  the  turnaround  time,  and  ranges  in  turnaround  time  can  be  de¬ 
fined  to  correspond  to  theoretical  levels  of  service.  In  a  similar 
manner,  criteria  can  be  established  for  other  parameters  that  affect 
how  useful  the  ECG  is  in  patient  care. 

3.  Development  of  Criteria 


In  order  to  establish  service  criteria  to  be  used  in  evaluation  of 
ECG  services,  it  was  important  to  have  a  thorough  knowledge  of  the  patient 
population  being  served,  and  the  different  uses  for  electrocardiograms 
within  the  military  health  care  system.  With  the  assistance  of  the  CAPOC 
representatives,  a  letter  was  sent  to  military  medical  personnel  in  the 
Army,  Navy  and  Air  Force,  seeking  their  views  on  standards  for  ECG  ser¬ 
vices  relating  to  the  four  service  parameters  identified: 

•  Turnaround  Time 

•  Use  of  Serial  Comparison 

•  Patient  Population  Receiving  ECGs 

•  Type  of  Reader 

Respondents  were  asked  to  provide  values  that  represented  acceptable 
services  and  preferred  services  for  different  types  of  patients  and  ECGs 
[e.g.  ,  acceptable  and  preferred  turnaround  times  were  requested  for  routine, 
STAT  (emergency)  and  pre-operative  inpatient  ECGs,  Emergency  Room  ECGs, 
and  diagnostic  and  other  outpatient  ECGs].  Respondents  were  also  asked 
for  their  views  of  other  important  factors  that  should  be  considered 
when  defining  preferred  and  acceptable  ECG  services.  The  request  also 
stated  that  the  purpose  of  the  criteria  was  to  specify  what  was  desirable — 
not  what  was  feasible,  or  what  existed  at  the  time.  A  total  of  55  re¬ 
sponses  were  received  and  used  in  establishing  criteria. 
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A  consensus  development  meeting  was  then  held  to  review  the  responses 
and  establish  the  criteria  that  would  be  used  in  the  evaluation.  The 
meeting  was  attended  by  at  least  one  cardiologist,  one  internist  or 
general  practitioner,  and  one  technician  from  each  of  the  three  mili¬ 
tary  services.  The  consensus  group  concluded  that  establishing  criteria 
for  patients  who  should  receive  ECGs  was  not  possible  and,  further,  that 
deciding  on  the  basis  of  statistical  information  whether  an  increase  in 
ECG  usage  for  a  given  patient  population  was  appropriate  or  beneficial 
would  be  impossible.  The  group  did  achieve  a  consensus  on  levels  for 
preferred  and  acceptable  services  for  the  remaining  three  parameters: 
turnaround  time,  type  of  reader,  and  use  of  serial  comparisons.  These 
criteria  are  shown  in  Table  2-1. 

For  each  type  of  ECG,  the  data  regarding  X  and  Y  Period  ECG  services 
at  the  evaluation  sites  were  compared  with  each  criterion  in  order  to 
determine  whether  preferred  of  acceptable  levels  of  ECG  services  were 
being  delivered.  In  order  for  an  ECG  to  be  considered  preferred,  it  had  to 
meet  aH  ECG  service  criteria  for  preferred  service.  In  order  for  an  ECG  to  be 
considered  acceptable,  it  had  to  either  meet  all  acceptable  criteria  or 
meet  preferred  or  acceptable  levels  for  all  criteria;  for  example,  if  an 

ECG  met  preferred  criteria  for  reader  and  serial  comparison  and  met 
acceptable  criteria  for  turnaround  time,  this  ECG  was  considered  acceptable. 
Together,  ECGs  meeting  preferred  criteria  and  acceptable  criteria  are  re¬ 
ferred  to  as  effective  ECGs. 

C.  DATA  COLLECTION 
1.  Quantitative  Data 


The  quantitative  portion  of  the  CAPOC  evaluation  required  in¬ 
formation  regarding  ECG  volumes  and  types,  service  effectiveness  as 
measured  against  the  evaluation  criteria,  and  the  costs  of  providing  ECG 
services.  These  data  were  collected  at  the  five  primary  evaluation  sites 
by  observation,  on-site  data  collection,  and  review  of  hospital  and  ECG 
department  records.  Table  2-2  summarizes  the  data  collection  techniques 
used  for  each  of  the  data  elements.  In  most  instances,  interviews  and 
and  direct  on-situ  observations  were  used  to  confirm  the  results  of  the 


data  collection. 
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TABLE  2-1 


CRITERIA  FOR  EVALUATING  THE  EFFECTIVENESS  OF  ECG  SERVICES 


Service  Parameter 

Acceptable  Service 

Preferred  Service 

What  do  you  consider  to  be  an 
acceptable  turnaround  time  for 

What  do  you  consider  to  be  the 
optimal  turnaround  time  to  get 

Turnaround  Time: 

the  ECG  interpretation  in  order 
to  be  useful  in  clinical 
management  of  the  patient? 

maximum  benefit  from  ECG  in¬ 
formation  in  clinical  decision¬ 
making? 

Inpatient  ECGs 

Routine 

2  days 

1  day 

STAT 

30  minutes 

15  minutes 

Pre-operative1 

24  hours 

12  hours 

Emergency  Room 

30  minutes 

15  minutes 

Oucpatient  ECGs 

Diagnostic  (for  thac  visit) 

30  minutes 

30  minutes 

(for  future  visit) 

3  days 

n/a 

Annual  Physical  and  Other 
Screening 

7  days 

2  days 

Training  of  ECG  Reader: 

What  staff  do  you  feel  can  pro¬ 
vide  "acceptable  ECG  reading" 

What  staff  do  you  feel  provide 
"optimal  ECG  reading"  for  the 

in  the  following  ECG  reading 
tasks? 

following  ECG  reading  tasks? 

In  a  Manual  Setting 

Classify  Normal  Tracings, 

Refer  Only  Abnormal  Tracings 
for  Reading 

Internist  or  Cardiologist 

Cardiologist 

Make  Initial  Reading  of  Normal 
and  Abnormal,  Refer  for 
Overreading 

Internist  or  Cardiologist 

Cardiologist 

Make  Final  Interpretation 
of  Normal  and  Abnormal  ECGs 

Internist  or  Cardiologist 

Cardiologist 

With  Computer-Assisted  Reading 

Certify  Normal  Interpretations 
Refer  Abnormal  Interpretations 
for  Overreading 

Corpsman/ECG  Technician, 

Nurse,  or  Any  Physician 

Corpsman/ECG  Technician, 

Nurse,  or  Any  Physician 

Overread  Computer  Interpretations  Internist  or  Cardiologist 

Cardiologist 

Availabilitv  of  a  Serial  Comparison:  At  a  minimum  which  ECGs  do 

In  an  optimal  system,  which 

you  feel  should  receive  a 
serial  comparison? 

ECGs  would  receive  a  serial 
comparison? 

All  Abnormal  ECGs 

All  Abnormal  ECGs 

For  pre-operative  ECGs  turnaround  time  was  defined  from  the  time  the  ECG  was  taken  to  the  time  that 
the  official  interpretation  was  available  in  the  patient's  chart.  For  other  ECGs  turnaround  time  was 
defined  from  the  time  the  ECG  was  taken  until  the  interpretation  was  returned  and  made  available  to 
the  care  provider. 
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Iqulpmrnt  Inventory  of  equipment  In  use.  Interviews  CAPOC  contract  documents,  inven-  Interviews 

Interviews  (age  of  equipment,  tory  of  equipment  In  use 

maintenance) 


A  series  of  data  collection  forms  was  developed  to  facilitate  re¬ 
cording  of  data.  Procedures  and  data  collection  instruments  were  pre¬ 
tested  at  Camp  Pendleton  NRMC  in  February  and  March  1979.  After  the 
pre-test,  the  data  collection  protocol  was  modified  to  accommodate  the 
specific  mode  of  operation  and  recordkeeping  practices  at  each  primary 
evaluation  site. 

2.  Qualitative  Information 

Collection  of  qualitative  information  was  accomplished  by  a  survey 
of  staff  and  staff  interviews  at  primary  evaluation  sites  and  by 
interviews  with  staff  at  the  secondary  evaluation  sites. 

Two  versions  of  a  staff  questionnaire  were  developed  for  use  before 
and  after  the  installation  of  CAPOC.  The  factors  to  be  evaluated  by  the 
qualitative  assessment  pertained  to  staff  acceptance  of  CAPOC,  changes 
in  patient  referral  patterns,  staff  satisfaction  with  ECG  services,  per¬ 
ceived  accuracy  of  interpretations,  relative  importance  of  attributes 
of  ECG  service,  and  time  devoted  to  performing  ECG  reading  functions. 

All  questionnaires  were  designed  so  that  they  could  be  filled  in 
without  assistance  from  other  staff  or  the  research  team.  Questionnaires 
were  also  designed  to  be  anonymous.  The  respondent's  date  of  birth  was 
used  as  an  identifier  so  that  responses  to  the  two  surveys  could  be 
compared. 

Questionnaires  and  methods  for  administering  them  were  pre-tested 
during  February  1979  at  Long  Beach  Naval  Regional  Medical  Center.  This 
site  was  selected  because  it  was  one  of  the  largest  hospitals  among  the 
secondary  evaluation  sites.  In  all,  16  staff  members  at  Long  Beach 
completed  the  staff  questionnaire  and  were  interviewed.  Professional 
staff  appeared  to  be  able  to  complete  the  questionnaires  easily.  They 
made  some  suggestions  regarding  terminology  that  were  incorporated  in  a 
revised  questionnaire  (see  Appendix  A  for  X  and  Y  Period  questionnaires). 

Obtaining  completed  questionnaires  from  busy  staff  members  was  the 
major  problem  identified.  This  pointed  out  the  necessity  for  having  a 
cover  letter  requesting  cooperation  signed  by  the  Chief  Medical  Officer 
to  accompany  the  questionnaire.  Therefore,  plans  were  made  to  contact 
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the  appropriate  staff  at  each  study  site  well  in  advance  of  the  visit 
to  arrange  for  the  production  of  such  a  letter,  to  obtain  assistance 
in  identifying  the  sample  population,  and  to  establish  procedures  for 
questionnaire  distribution  and  return. 

Information  collected  at  the  secondary  sites  was  largely  qualitative 
in  nature,  obtained  through  interviews  with  staff  involved  with  ECG 
services.  So  that  consistent  information  was  obtained  at  each  site, 
a  detailed  interview  guide  was  prepared  and  used  (see  Appendix  B  for 
X  and  Y  Period  interview  guides).  At  some  of  the  secondary  sites  actual 
data  concerning  ECG  volumes  were  obtained  and  in  a  few  cases  a  limited 
amount  of  original  data  collection  was  undertaken. 

D.  DATA  ANALYSIS 
1.  Quantitative  Data 
a.  ECG  Volumes 

Information  regarding  ECG  volumes  was  necessary  for  projecting  the 
measured  service  effectiveness  to  all  ECG  services  and  for  computing  the 
costs  per  ECG  processed  in  each  site  and  in  the  region.  Because  of  dif¬ 
ferent  recordkeeping  practices  in  the  different  sites  and  the  importance 
of  the  number  of  ECGs  in  the  calculation  of  cost-effectiveness  and  the 
conclusions,  X  and  Y  Period  volume  data  for  each  site  were  scrutinized 
for  reasonableness  and  major  discrepancies  or  changes  in  volumes  between 
the  X  and  Y  Periods  were  explored.  In  one  case  (March  AFB)  the  baseline 
data  were  adjusted  to  reflect  an  annual  volume  of  6,000  rather  than  10,608 
ECGs  where  it  was  discovered  that  data  in  the  baseline  reflected  duplicate 
tracings  taken  on  the  same  patient. 

In  some  cases,  portions  of  the  ECG  volume  for  a  given  site  had  to 
be  estimated  from  a  combination  of  data  sources.  This  was  particularly 
true  for  X  Period  data,  which  had  to  be  reconstructed  from  only  avail¬ 
able  records.  The  actual  volumes  used  in  the  analysis  are  based  upon 
best  judgments  of  reasonable  values. 


The  data  revealed  an  11%  drop  in  ECG  volume  in  the  region  from  the  X 
Period  (1979)  to  the  Y  Period  (1980).  Though  the  cause  of  this  change 
has  not  been  identified,  there  are  several  plausible  explanations: 
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(1)  The  baseline  data  are  in  error.  Since  baseline  data  were 
collected  from  manual  records,  and  in  many  cases,  Y  Period 
data  were  automatically  collected  by  the  system,  the  manual 
records  may  reflect  estimates  instead  of  accurate  counts  of 
ECGs  taken. 

(2)  The  CAPOC  system  lead  to  fewer  repeat  duplicate  tracings 
because  of  lost  or  delayed  interpretations  or  poor  quality 
tracings.  Most  physicians  surveyed  felt  that  repeat  duplicate 
tracings  had  declined  as  a  result  of  CAPOC;  however,  there  are 
no  data  to  confirm  the  magnitude  of  this  impact. 

(3)  The  volume  actually  did  decrease  due  to  changes  in  patient  popu¬ 
lation  or  medical  practice.  (It  is  interesting  to  note  that  data 
from  surveys  of  civilian  ambulatory  medical  care  show  a  decline 
of  ECG  volume  of  24%  between  1978  and  1979.) 

The  initial  analysis  was  performed  with  the  volume  data  as  collected. 
If  the  system  resulted  in  a  reduction  in  duplicate  tracings  and/or  the 
volume  did  decline,  the  CAPOC  system  would  be  more  cost-effective  than 
the  analysis  indicates. 

The  impact  of  the  CAPOC  system  on  effectiveness  of  services  also 
depends  upon  the  mix  of  normal  and  abnormal  ECGs  (since  different  types 
of  physicians  are  considered  effective  and  preferred  readers  for  normal 

and  abnormal  tracings).  The  effective  turnaround  time  is  dependent 
on  the  type  of  patient  involved,  for  example,  pre-operative  ECGs  are 
required  sooner  than  routine  outpatient  ECGs.  Therefore,  the  mix  of 
patients  receiving  ECG  services  had  to  be  determined  accurately  in  the 
evaluation. 

At  the  secondary  evaluation  sites,  the  type  of  ECGs  taken  in  the 
baseline  period  were  estimated.  After  CAPOC  implementation,  often 
better  records  (or  different  estimates)  of  the  types  of  ECGs  were  ob¬ 
tained.  Where  the  differences  were  large,  the  sensitivity  of  conclusions 
to  different  assumptions  was  examined. 

b.  Service  Performance  Measured  Against  Evaluation  Criteria 

The  quantitative  evaluation  was  focused  on  determining  the  annual 
number  of  ECGs  at  each  site  meeting  criteria  for  acceptable  and  preferred 
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TABLE  2-3  DATA  CONCERNING  INDIVIDUAL  SERVICE  CRITERIA  IN  THE  X  PERIOD  AT 
SAN  DIEGO  NAVAL  REGIONAL  MEDICAL  CENTER 
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ECG  services.  The  basic  approach  was  to  obtain  data  on  a  sample  of  ECGs 
and  project  the  results  to  the  annual  volumes. 

An  example  of  how  this  was  accomplished  is  shown  in  Tables  2-3,  2-4,  and 
2-5  for  X  Period  data  for  San  Diego  Naval  Regional  Medical  Center.  In 
the  first  table,  the  quantitative  data  on  turnaround  time,  ECG  reader, 
and  serial  comparison  are  summarized  for  normal  and  abnormal  ECGS  broken 
down  into  five  service  categories: 
routine  outpatient 
annual  physical 
routine  inpatient 
pre-operative 
STAT/Emergency  Room 

Each  of  these  service  categories  must  be  considered  separately  because 
the  criteria  for  turnaround  time  are  different  for  each.  Likewise, 
abnormal  and  normal  ECGs  must  also  be  Considered  separately  because  the 
criteria  for  reader  and  serial  comparison  differ. 

All  data  shown  in  Table  2-3  are  percentages  of  a  sample  of  the  ECG 
volume  (at  San  Diego  NRMC  902  ECGs  and  865  ECGs  were  sampled  in  the  X 
and  Y  Periods  respectively).  As  noted  under  the  column  "Data  Source" 
for  each  service  category,  data  for  San  Diego  NRMC  were  all  actual 
data  reported  for  or  measured  at  the  site;  for  certain  data  at  other 
primary  sites  and  for  most  data  at  secondary  sites,  data  were  obtained 
in  interviews  or  estimated  based  upon  information  from  a  similar  site. 

Once  the  data  on  services  had  been  complied,  they  were  compared  with 
the  evaluation  criteria  in  order  to  analyze  what  percentage  of  ECGs  pro¬ 
cessed  at  the  site  meet  acceptable  and  preferred  service  criteria.  The 
manner  in  which  this  was  accomplished  is  shown  in  Table  2-4  for  normal 
ECGs  and  Table  2-5  for  abnormal  ECGs ;  again  the  data  are  for  the  X  Period 
at  San  Diego  NRMC. 

For  each  type  of  ECG,  the  percentages  meeting  acceptable  and  pre¬ 
ferred  criteria  for  turnaround  time,  ECG  reader,  and  serial  comparison  were 
computed.  ECGs  had  to  meet  all  criteria  for  preferred  services  in  order 
to  be  considered  preferred  ECGs;  they  had  to  be  rated  as  acceptable  or 
preferred  for  each  criterion  in  order  to  be  considered  effective  ECGs. 
Percentages  of  ECGs  meeting  the  three  criteria  were  applied  in  a  step- 
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wise  fashion  in  order  to  compute  the  percentage  of  each  type  and  of  the 
total  volume  that  met  criteria  for  acceptable  and  preferred  services. 

The  following  example  of  this  procedure  is  taken  from  Tables  2-4  and 
2-5  and  concerns  routine  inpatient  ECGs.  As  noted  in  Table  2-3,  56%  of 
these  ECGs  were  normal.  Of  all  normal,  routine  inpatient  ECGs,  58%  were 
returned  within  1  day  and  a  total  of  92%  (58%  +  34%)  were  returned  within 
2  days.  Hence  58%  met  the  criterion  for  preferred  turnaround  time  and  92% 
met  the  criterion  for  effective  (preferred  plus  acceptable)  turnaround  time. 

For  this  site,  87%  of  the  normal,  routine  inpatient  ECGs  are  read  by 
a  cardiologist  (a  preferred  reader  according  to  the  evaluation  criterion). 
All  (100%)  of  these  ECGs  are  read  by  an  effective  reader. 

The  percentage  meeting  all  criteria  is  the  product  meeting  each 
separate  criterion.  Thus  the  percent  meeting  preferred  criteria  is 

58%  (turnaround  time)  x  87%  (reader)  =  50.5%  (all  criteria). 

Normal  ECGs  account  for  37%  of  all  routine  inpatient  ECGs  which,  in  turn, 
account  for  42%  of  the  total  volume.  Thus,  normal,  routine  inpatient 
ECGs  contribute: 

50.5%  (all  criteria)  x  37%  (normal)  x  42%  (routine  inpatient)  = 

7.8%  to  the  total  preferred  level  of  service. 

The  criteria  for  serial  comparison  must  be  included  in  the  calcula¬ 
tion  of  abnormal  ECGs  meeting  all  the  criteria.  As  seen  in  Table  2-3, 

46%  of  the  abnormal,  routine  inpatient  ECGs  receive  a  serial  comparison. 

The  calculation  of  percent  of  these  ECGs  meeting  preferred  criteria  is: 

58%  (turnaround  time)  x  85%  (reader)  x  46%  (serial  comparison)  =  22.7%. 

This  number  is  then  carried  through  in  the  same  manner  as  the  calculation 
for  the  normal  ECGs. 

Preferred  ECGs  meet  all  the  criteria  for  preferred  level  of  service. 
Effective  ECGs  are  the  sum  of  ECGs  meeting  preferred  service  criteria  and 
of  ECGs  meeting  acceptable,  but  not  preferred,  services.  In  subsequent 
discussions  and  tables,  the  term  effective  ECG  refers  to  this  sum.  Accept¬ 
able  ECGs  can  be  determined  in  the  calculation  by  taking  the  appropriate 
difference. 
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This  same  procedure  was  followed  for  each  of  the  18  evaluation  sites 
for  both  the  X  Period  and  the  Y  Period  (a  computer  program  was  eventually 
developed  for  performing  the  calculations).  At  some  sites  where  CAPOC 
was  not  yet  capturing  the  entire  ECG  volume  when  Y  Period  data  were  col¬ 
lected,  two  analyses  were  performed:  one  calculating  the  effectiveness 
of  the  CAPOC  portion  of  the  volume  and  one  calculating  the  effectiveness 
of  the  manual  portion  of  the  volume.  In  addition,  system  performance 
when  CAPOC  is  fully  implemented  was  projected  and  the  resulting  service 
effectiveness  calculated. 

c.  Cost  Analysis 
(i)  Introduction 

Measurements  of  costs  of  ECG  services  in  the  X  and  Y  Periods  were 
essential  in  order  to  determine  the  cost-effectiveness  of  manual  vs. 

CAPOC  services.  Five  separate  direct  cost  elements  were  identified  and 
used  in  the  analysis: 

•  labor  costs, 

•  supply  costs, 

•  capital  equipment  costs, 

0  maintenance  costs,  and 

•  system  costs  (Y  Period  only). 

Overhead  costs  were  considered  initially.  However,  because  information 
on  overhead  cost  allocations  was  available  for  only  one  site  (Camp 
Pendleton),  these  costs  were  not  included  in  the  initial  cost  calculations. 
The  effect  of  the  overhead  is  considered  in  the  sensitivity  analysis. 

Because  of  data  limitations,  several  different  methods  were  utilized 
to  compile  the  costs.  When  available,  actual  data  from  the  site  were 
used  in  arriving  at  the  costs  for  the  individual  elements.  If  data  were 
unavailable  for  a  site,  data  from  another  site  that  was  determined  to  be 
similar  in  such  areas  as  ECG  process,  size,  and  volume  were  scaled  and/or 
adjusted  as  required  in  order  to  obtain  an  estimate  for  the  site. 

The  Y  Period  analysis  consisted  primarily  of  a  differential  cost 
analysis.  Differential  costs  are  those  costs  that  resulted  from  the  instal¬ 
lation  of  CAPOC.  In  order  to  perform  the  differential  cost  analysis 
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each  element  is  examined  in  order  to  determine  additional  costs,  savings, 
or  trade-offs  resulted  from  use  of  the  system.  These  are  then 
added  to  the  X  Period  (manual  system)  costs  to  estimate  the  costs  of 
services  with  CAPOC. 

Specific  methods  and  data  used  for  each  data  element  are  summarized 
in  the  following  section. 

(ii)  Cost  Elements 
(a)  Labor  Costs 

Labor  costs  are  comprised  of  two  components: 

•  the  cost  for  the  physician  time  devoted  to  reading 
the  ECG,  and 

e  the  cost  for  the  ECG  technician  time  spent  performing 
the  various  functions  involved  in  the  overall  ECG 
process. 

Thus  calculating  the  labor  costs  required  information  about  the  cost  of 
labor  and  the  time  devoted  to  the  ECG  process. 

Labor  costs  were  based  on  actual  average  pay  rates  for  the  various 
personnel  concerned.  The  average  base  salary  was  multiplied  by  1.38  to 
reflect  the  value  of  medical  and  retirement  benefits.  This  product  was 
in  turn  multiplied  by  1.209  to  reflect  leave  and  holiday  pay.  Thus, 
base  salaries  were  adjusted  by  a  factor  of  1.668.  In  order  to  obtain 
the  composite  salary,  any  allowance  for  quarters  or  other  special  pay¬ 
ments  are  added  to  the  adjusted  salary.  A  cost  per  hour  was  obtained  by 
dividing  the  composite  salary  by  2080  hours.  A  summary  of  labor  costs 
for  all  categories  of  personnel  is  presented  in  Table  2-6. 

Several  different  methods  were  employed  to  determine  the  time  spent 
by  the  various  staff  types  involved  in  the  ECG  process.  Reading  times 
for  normal  and  abnormal  ECGs  or  for  all  ECG  were  obtained  by  direct 
observation,  through  questionnaires,  and  by  interviews.  Where  necessary, 
data  from  a  similar  site  were  used  for  a  site  were  no  data  were  available. 
Data  on  type  of  reader — i.e.,  cardiologist,  internist,  general  practioner, 
or  resident — were  obtained  from  interviews  or  review  of  ECG  interpreta¬ 
tions.  These  data  were  combined  with  the  volume  of  ECGs  for  each  site 
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TABLE 

2-6 

SUMMARY 

OF  HOURLY 

LABOR  COSTS* 

Staff  Type 

Typical 

Pay 

Grades 

Base 

Salary 

Quarters 

and/or 

Other 

Allowances 

Composite 

Salary 

Cardiologist 

5-6 

$29,987 

$10,933 

$60,964 

$29.31 

Internist 

5-6 

$29,987 

$10,933 

$60,964 

$29.31 

General 

Practitioner 

3-4 

$20,806 

$  7,623 

$42,336 

$20.35 

Resident 

3 

$18,894 

$  7,283 

$38,806 

$18.66 

Technician 

E-4 

$  7,408 

$  3,373 

$15,733 

$  7.56 

E-3 

$  6,480 

$  2,728 

$13,539 

$  6.51 

*Salaries  are 

in  1980 

dollars . 
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in  order  to  calculate  total  reading  cost.  The  general  overall  calculation 
of  reading  cost  for  a  site  may  be  summarized  as  follows: 


z 

N  X  RiN  X  TN  X  Ci.  +  E 

A  x  R.  xT.  xC. 
iA  A  1 

All  normal 

All  abnormal 

ECGs 

ECGs 

where : 

N  =  annual  volume  of  normal  ECGs  at  the  site, 

R  =  percent  of  normal  ECGs  read  by  reader  type  i. 


=  time  required  to  read  one  normal  ECG, 

=  labor  cost  for  reader  type  i, 

A  =  annual  volume  of  abnormal  ECGs  at  the  site, 

R  =  percent  of  abnormal  ECGs  read  by  reader  type  i, 

1 A 

T  =  time  required  to  read  one  abnormal  ECG,  and 
A 

i  =  reader  type:  cardiologist,  internist,  general 
practitioner,  resident. 

The  differential  costs  due  to  the  installation  of  CAPOC  were  determined 
in  the  Y  Period  by  multiplying  the  difference  in  reading  time  per  ECG  by 
the  appropriate  cost  per  unit  time  and  annual  ECG  volume.  This  yielded 
the  differential  cost  for  reading  labor. 

Two  different  approaches  were  used  to  calculate  technician  costs 
for  processing  ECGs.  At  sites  where  one  or  more  technicians  were  assigned 
to  handle  ECGs  exclusively,  the  labor  cost  was  determined  by  multiplying 
the  number  of  technicians  by  the  average  annual  composite  salary.  In 
sites  where  technicians  had  other  duties  not  directly  related  to  ECGs, 
an  average  time  per  ECG  was  obtained.  These  values  were  either  measured 
or  were  estimated  through  interviews.  The  average  time  per  ECG  was 
multiplied  by  the  annual  ECG  volume  and  the  unit  cost  of  labor  to  deter¬ 
mine  the  total  annual  technician  labor  cost. 

A  marginal  cost  for  technician  labor  was  calculated  for  the  Y  Period 
based  upon  change  in  time  and/or  elimination  of  an  activity  due  to 
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CAPOC.  The  time  per  ECG  was  multiplied  by  the  annual  ECG  volume  and  the 
cost  per  unit  time  to  yield  the  total  difference. 

(b)  Supply  Costs 

Two  methods  were  used  to  estimate  supply  costs  in  the  X  Period. 
Wherever  possible,  supply  costs  were  based  on  actual  cost  per  item  and 
actual  total  number  of  items  ordered  in  a  one-year  period. 

At  some  sites,  however,  the  data  on  supply  usage  and  costs  were 
not  available.  For  these  sites  a  cost  per  ECG  was  calculated  on  the 
basis  of  knowledge  of  the  ECG  process  and  data  on  average  unit  cost  for 
each  item  used  (obtained  from  sites  where  information  was  available). 

The  cost  per  ECG  was  multiplied  by  the  annual  volume  to  yield  the  annual 
supply  costs  for  a  facility. 

In  the  Y  Period  analysis,  it  was  necessary  to  identify  both  in¬ 
creased  costs  due  to  such  factors  as  more  expensive  paper  and  cost 
savings  due  to  elimination  or  reduction  of  certain  supplies.  All  cost 
differences  were  based  upon  unit  costs  per  ECG  multiplied  by  the  annual 
ECG  volume. 

(c)  Capital  Equipment  Costs 

Capital  equipment  costs  for  the  X  Period  were  limited  to  ECG  cart 
costs.  The  age  of  the  ECG  carts  and  purchase  prices  varied  widely.  The 
useful  lifetime  for  this  type  of  equipment  is  generally  considered  to  be 
5  years  (three- channel  transmitting  carts)  or  7  years  (single-channel  or 
three-channel  non-transmitting  carts).  A  number  of  carts  in  use  in  the 
18  CAPOC  sites  had  exceeded  their  useful  lifetime  for  depreciation 
purposes,  and  therefore,  might  be  considered  to  have  no  cost  associated 
with  their  continued  use.  Probably,  these  carts  should  have  been  re¬ 
placed  (the  unreliability  of  outdated  equipment  was  a  recurring  physician 
comment  at  many  sites). 

However,  in  order  to  assign  a  value  to  all  equipment  for  the  purposes 
of  this  evaluation,  an  average  annual  cost  was  calculated  for  each  type 
of  cart,  i.e. ,  single-channel,  three-channel,  non-transmitting,  and  three- 
channel  transmitting,  based  on  actual  purchase  prices  and  assumed  lifetimes 
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of  5  years  or  7  years.  The  cost  range,  average  costs,  and  lifetime  and 
annual  cost  for  each  piece  of  equipment  are  summarized  in  Table  2-7. 

This  average  annual  cost  was  assigned  to  all  carts  irrespective  of  their 
age.  (Y  Period  capital  costs  are  discussed  under  system  costs.) 

(d)  Maintenance  Costs 

Equipment  maintenance  costs  were  of  two  types:  fixed-price  contracts 
and  maintenance  performed  by  the  site  staff.  When  the  equipment  was 
covered  by  a  maintenance  contract  (six  sites)  ,  the  annual  contract  price 
was  used.  Three  sites  maintained  records  on  maintenance  and  repair  time 
for  the  ECG  equipment.  From  these  data,  an  average  annual  maintenance 
time  for  a  single-channel  and  a  three-channel  cart  was  calculated. 

These  times  were  converted  to  an  annual  cost  using  the  pay  rate  of  an  E-4 
technician.  These  average  costs  were  then  applied  to  the  eight  sites 
where  data  were  not  available.  The  average  annual  maintenance  cost  for 
a  single-channel  ECG  was  $38,  for  a  three-channel  ECG  cart,  $71. 

(e)  System  Cost  (Y  Period  Only) 

System  costs  were  obtained  from  the  contract  negotiated  with  the 
vendor.  Five  cost  elements  were  specifically  identified  in  the  contract: 

•  initial  costs — including  training  and  software  conversion, 

•  lease  payments , 

•  equipment  purchase  price, 

•  maintenance  and  software  costs,  and 

•  communication  costs. 

In  addition  to  these  costs,  other  costs  were  incurred  by  the  sites  for 
the  purchase  of  additional  equipment  and  additional  supplies.  Site- 
specific  costs  were  obtained  from  the  sites.  These  two  categories  of 
cost  comprise  the  expenses  directly  attributable  to  the  CAPOC  system. 

(iii)  Life-Cycle  Cost  Calculations 

In  order  to  calculate  an  average  annual  cost,  a  life-cycle  cost 
calculation  was  performed.  The  CAPOC  system  life  was  taken  to  be  8  years. 
For  each  year  in  the  system  life,  the  expenses  and  savings  were  calculated 
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TABLE  2-7 


EQUIPMENT  COSTS 


Type  of  Cart 

Average 

Cost 

Lifetime 

Single- Channel 

$1,308 

7 

years 

Three-Channel , 
Non-Transmitting 

$5,762 

7 

years 

Three- Channel , 
Transmitting 

$7,788 

5 

years 

Average  Annual 
Cost. _ 

$  187 

$  823 

$1,558 
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by  one  of  several  methods  depending  upon  the  cost  elements.  Initial  and 
equipment  costs  (lease  then  purchase)  were  specified  in  the  contract  with 
the  CAPOC  vendor.  Maintenance/software  and  communication  costs  were 
specified  in  the  contract  and  included  an  escalation  factor  of  10  percent 
annually  after  the  second  year.  Paper  costs  were  escalated  at  10  percent 
annually  after  the  first  year.  Savings  were  also  escalated.  Differences 
in  paper  and  supply  costs  were  escalated  at  10  percent  annually  after  the 
first  year.  Labor  savings  were  escalated  at  varying  yearly  rates 
according  to  a  DoD  schedule  of  projected  labor  cost  increases. 

For  each  year  of  system  life  the  sum  of  expenses  and  the  sum  of 
savings  were  then  discounted  at  10  percent  in  order  to  obtain  a  net 
present  value.  The  net  present  value  of  costs  minus  the  total  net  present 
value  of  the  savings  gave  the  total  system  life-cycle  cost.  The  total 
system  life-cycle  cost  divided  by  the  system  life  time  yielded  the  in¬ 
cremental  annual  cost  of  the  CAPOC  system,  which  was  used  in  the  cost- 

effectiveness  analysis. 

(d)  Cost-Effectiveness  Analysis 

The  cost  effectiveness  of  ECG  services  was  obtained  by  dividing  the 
total  annual  cost  of  service  by  the  number  of  effective  ECGs  to  yield 
a  cost  per  effective  ECG.  Actually,  two  cost  figures  were  calculated  in 
each  period.  A  cost  per  effective  ECG  (the  sum  of  ECGs  meeting  accept¬ 
able  and  preferred  criteria)  and  a  cost  per  ECG  at  the  preferred  level  of 
service.  The  X  Period  costs  were  the  summation  of  each  cost  time  dis¬ 
cussed  previously.  The  Y  Period  costs  were  the  sura  of  the  X  Period  cost 
plus  the  incremental  annual  cost  due  to  CAPOC. 

(e)  Sensitivity  Analysis 

In  performing  the  analysis  of  service  effectiveness,  several  assump¬ 
tions  were  required  concerning  various  aspects  of  the  ECG  process.  For 
example,  certain  portions  of  the  costs,  specifically  the  variable  costs, 
are  directly  related  to  the  number  of  ECGs  processed.  Several  different 
scenarios  were  evaluated  in  order  to  estimate  the  magnitude  of  the  effect 
of  major  assumptions  on  the  conclusions  about  cost-effectiveness. 
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2.  Qualitative  Information 

Questionnaire  results  for  the  X  and  Y  Periods  were  analyzed  from 
data  tabulations  prepared  with  the  aid  of  a  computer  program.  Informa¬ 
tion  relating  to  desired  turnaround  times  and  time  devoted  to  ECG  reading 
was  plotted  on  a  distribution  chart.  Responses  regarding  the  importance 
of  attributes  of  ECG  service  and  satisfaction  were  weighted  (e.g.,  +2 
for  very  good,  +1  for  good,  etc.)  and  tabulated  so  that  they  could  be 
compared  more  readily. 

Many  of  the  survey  questions  were  related  to  the  service  effective¬ 
ness  criteria  used  in  the  quantitative  portion  of  the  evaluation.  The 
responses  tc  these  questions  were  analyzed  in  order  to  identify  areas 
of  agreement  or  disagreement  with  the  evaluation  criteria  and  the  quan¬ 
titative  findings  regarding  service  effectiveness. 

The  questionnaire  also  allowed  for  fill-in  responses  and  free-form 
comments  about  any  aspect  of  ECG  services.  These  provided  insight  into 
problems  or  special  conditions  at  each  primary  site.  This  information 
was  combined  with  the  results  of  interviews  conducted  during  site  visits 
to  reach  conclusions  regarding  the  experience  with  CAPOC  implementation 
at  each  site. 
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CHAPTER  3.  SERVICE  EFFECTIVENESS— PRIMARY  SITES 


A.  SAN  DIEGO  NAVAL  REGIONAL  MEDICAL  CENTER 


1.  Introduction 


San  Diego  Naval  Regional  Medical  Center  is  the  largest  military 
treatment  facility  with  the  largest  cardiology  staff  and  the  most  compre¬ 
hensive  diagnostic  and  treatment  services  in  the  San  Diego  region.  An 
average  of  2,318  ECGs  per  month  were  taken  at  this  hospital  in  FY  1979 
(X  Period) ;  this  represented  32  percent  of  the  volume  of  ECGs  taken  at 
facilities  to  be  included  in  CAPOC  in  the  Southern  California  area. 

In  calendar  year  1980,  San  Diego  NRMC  averaged  2,002  ECGs  per  month, 
a  decrease  of  13.6  percent  from  1979.  This  represented  32  percent  of 
all  ECGs  done  at  facilities  in  the  San  Diego  region  that  have  (or  will 
have)  access  to  CAPOC. 

In  addition  to  those  ECGs  taken  at  the  hospital,  San  Diego  NRMC 
provides  reading  services  for  branch  clinics.  These  six  branch  clinics 
that  would  have  access  to  CAPOC  averaged  918  ECGs  per  month  in  1979. 

This  number  decreased  slightly  to  910  ECGs  per  month  in  1980  for  the 
same  size  branch  clinics.  This  means  that  in  1979,  48.6  percent  of  all 
ECGs  done  in  the  Southern  California  area  were  interpreted  at  San  Diego  NRMC. 
This  number  decreased  to  46.3  percent  in  the  Y  Period. 

Because  of  its  role  in  providing  comprehensive  patient  care,  and 
ECG  services  in  particular,  and  its  role  as  the  central  site  in  the  CAPOC 
project,  San  Diego  NRMC  was  a  major  data  collection  site  for  the  evalua¬ 
tion. 

San  Diego  NRMC  consists  of  a  large  hospital  and  12  branch  clinics. 

Six  of  the  branch  clinics  have  (or  will  have)  access  to  CAPOC.  Data 
on  these  sites — one  primary  evaluation  site  and  five  secondary  sites — are 
included  in  this  report.  This  section  discusses  the  activities  at  San 
Diego  NRMC  and  those  activities  that  are  relative  to  the  branch  clinics. 

San  Diego  NRMC  has  approximately  600  authorized  beds  in  the  hospital 
itself.  The  occupancy  averaged  472  beds  in  1979  and  463  beds  in  1980. 
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In  1979  over  712,000  outpatient  visits  were  recorded;  this  number  rose 
to  886,000  in  1980.  In  total,  the  hospital  and  12  branch  clinics  had 
1.3  million  outpatient  visits  in  1979  and  1.6  million  in  1980. 

The  hospital  staff  included  479  physicians  in  1979  and  488  in  1980. 
Residents  and  interns  totaled  286  in  1979  and  282  in  1980.  Total  military 
personnel  at  San  Diego  NRMC  were  1818  in  1979  and  1915  in  1980.  The 
Cardiology  Service  included  four  cardiologists,  two  pediatric  cardiolo¬ 
gists,  six  fellows  in  cardiology,  and  from  two  to  five  residents  and 
interns.  Between  1979  and  1980  data  collection  periods,  two  cardiologists 
and  one  cardiology  fellow  left  and  were  replaced  by  one  cardiologist  and 
two  cardiology  fellows.  The  cardiology  staff  shares  the  bulk  of  the 
ECG  reading  responsibilities.  In  this  text  the  term  "cardiologist  (s)" 
refers  to  the  staff  of  the  Cardiology  Service,  including  fellows  and 
house  staff,  who  read  ECGs,  unless  otherwise  specified. 

The  staff  of  the  hospital  of  San  Diego  NRMC  refers  to  the  hospital 
only  as  "NRMC"  in  spite  of  the  formal  organization  that  includes  the 
hospital  plus  12  branch  clinics.  This  report  also  refers  to  the  hospital 
as  San  Diego  NRMC  and  to  the  specific  branch  clinics  if  they  are  intended 
to  be  included  in  any  discussion. 


ECG  Service  Organization  and  Staffing 


The  Heart  Station  at  San  Diego  NRMC  is  the  primary  focus  for  ECG 

services.  The  manual  ECG  process  as  observed  in  September  of  1979  is 

shown  in  Figure  3-1.  The  average  time  for  each  part  of  the  process  is 

included  in  Figure  3-1.  These  values  were  obtained  by  direct  observation. 

The  process  was  initiated  when  a  request  for  an  ECG  is  filled  out  by  a 

nurse  or  corpsman.  If  the  patient  was  anihulatory^  he  proceeded  with  the 

7 

request  to  the  Heart  Station.  At  the  Heart  Station  information  was 
entered  in  a  log  and  cover  sheets  were  prepared  and  attached  to  the  re¬ 
quisition. 

~  .  —  ~~  "" 

61.4  percent  of  ECG  patients  were  ambulatory. 

^The  Heart  Station  operates  from  0730-1130  and  1230-1630,  Jlonday-Friday . 
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FIGURE  3-1  FLOWCHART  OF  THE  ECG  PROCESS  AT  SAN  DIEGO  NAVAL  REGIONAL  MEDICAL  CENTER 
AND  AVERAGE  TIMES  SPENT  OR  ELAPSED  FOR  INDIVIDUAL  STEPS,  SEPTEMBER  1979 


The  patient  was  asked  if  he  had  had  a  previous  ECG  at  San  Diego 

g 

NRMC.  If  the  patient  had  had  a  prior  ECG,  the  corpsman  retrieved  the  old 
ECG(s)  from  the  files  maintained  in  the  Heart  Station.  The  ECG  file 
was  attached  to  the  request  and  cover  sheet  and  the  patient's  ECG  was 
taken.  Two  tracings  were  made — one  was  given  to  the  patient  to  take 
with  him  and  the  other  kept  in  the  Heart  Station  to  be  read. 

Tracings  were  obtained  for  patients  who  were  not  ambulatory  when  a 
corpsman  from  the  Heart  Station  made  the  rounds  of  the  wards.  For  each 
ward  the  requests  were  found  in  a  box  at  the  nursing  station.  Two  trac¬ 
ings  were  recorded;  one  was  left  in  the  patient's  chart  and  one  was 
brought  to  the  Heart  Station.  There  the  technician  entered  the  necessary 
data  in  the  log,  retrieved  prior  ECGs  from  the  file,  and  then  left  the 
ECGs  for  reading. 

ECGs  were  also  taken  in  the  Emergency  Room  and  the  intensive  care 
units  by  the  personnel  assigned  to  these  units.  These  tracings  were  given 
initial  readings  by  the  internist  or  cardiologist  caring  for  the  patient 
and  then  sent  to  the  Heart  Station  to  be  logged  in,  formally  read,  and 
filed.  STAT  ECGs  for  inpatients  were  done  by  a  corpsman  from  the  Heart 
Station  and  were  left  in  the  patient's  chart  to  be  read  by  the  requesting 
physician. 

All  ECGs  for  pre-operative  patients  were  read  by  the  cardiologist 
assigned  to  the  Heart  Station  to  perform  readings  that  day;  these  ECGs 
were  read  after  1630  of  the  day  the  tracings  were  taken.  The  corpsman  on 
duty  was  responsible  for  telephoning  the  interpretation  to  the  appropriate 
ward  that  evening.  He  also  recorded  the  patient's  name  and  location  and 
the  ECG  interpretation  in  the  watch  log.  The  personnel  on  the  wards 
were  responsible  for  writing  the  telephoned  interpretation  in  the  patient's 
chart.  All  other  ECGs  were  read  by  the  cardiologists  who  did  the  bulk 
of  the  reading  the  following  morning.  The  interpretation  was  handwritten 
on  the  requisition  form  and  signed  by  the  cardiologist. 

After  the  ECGs  had  been  read,  the  handwritten  interpretation  was 
photocopied  and  returned  via  the  mail  to  the  location  from  which  the 

%0. 4  percent  of  patients  who  had  had  ECGs  done  in  the  Heart  Station  responded 
"yes"  to  this  question. 
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patient  came.  Eventually  this  copy  was  filed  in  the  patient's  medical 
record.  The  original  copy  of  the  interpretation,  along  with  the  tracing, 
was  filed  at  the  Heart  Station.  A  folder  and  label  were  prepared  for 
those  ECGs  for  patients  with  no  prior  tracing.  The  filing  step  completed 
the  ECG  process  as  it  was  in  place  during  the  X  Period  data  collection 
in  September  of  1979. 

Several  changes  have  occurred  as  a  result  of  CAPOC  implementation 
at  San  Diego  NRMC.  The  revised  ECG  process  when  CAPOC  is  fully  imple¬ 
mented  (i.e.,  automatic  storage  and  retrieval)  is  shown  in  Figure  3-2. 

The  relevant  times  observed  during  the  Y  Period  data  collection  are  pre¬ 
sented  in  Figure  3-2.  The  following  discussion  highlights  the  changes 
in  the  process  that  have  occurred  as  a  result  of  CAPOC  and  were  observed 
during  the  Y  Period  data  collection  effort. 

For  patients  having  ECGs  in  the  Heart  Station,  the  ECG  technicians 
prepared  one  cover  sheet  instead  of  two.  The  log  was  kept  only  as  a 
reference  for  any  inquiries  about  patients  being  sent  for  an  ECG.  The 
log  was  a  record  of  only  the  ECGs  done  in  the  Heart  Station  and  was 
discarded  at  the  end  of  the  month.  Previously,  logs  were  kept  for  one 
year  and  were  used  to  prepare  the  necessary  reports  to  the  Commanding 
Officer.  All  log  data  are  now  maintained  by  CAPOC. 

The  ECG  technician  made  only  one  tracing  in  the  Y  Period,  instead 
of  two.  This  tracing  remained  with  the  patient  while  the  ECG  request 
was  matched  with  the  unconfirmed  interpretation  generated  at  the  over¬ 
read  site  in  the  Heart  Station.  The  time  to  obtain  the  ECG  has  not 
changed  significantly  since  the  time  required  for  the  additional  tracing  has 
been  replaced  by  the  time  required  to  enter  patient  data  in  the  system. 

The  matching  of  the  ECG  request  with  the  unconfirmed  interpretation  was 
a  new  activity  resulting  from  CAPOC. 

The  ECG  reading  process  remained  unchanged.  ECGs  for  pre-operative 
patients  were  read  in  the  evening  and  the  results  called  back  to  the 
floor.  All  other  ECGs  were  read  the  next  morning.  Readers  entered 
changes  to  the  computer-generated  interpretation  by  crossing  out  comments, 
adding  standard  comments  using  number  codes,  or  adding  free-form  text 
comments.  The  reader  put  his  "number"  on  the  interpretation  when  he 
was  finished. 
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Following  the  morning  reading  session,  all  ECGs  were  edited  at  the 
overread  station  in  the  Heart  Station.  For  those  ECGs  done  at  San  Diego 
NRMC,  a  copy  of  the  tracing  with  the  confirmed  interpretation  was  genera¬ 
ted.  This  ECG  was  matched  with  the  ECG  request  and  returned  to  the 
patient's  medical  record  via  the  mail.  ECGs  from  the  branch  clinic 
were  edited  and  the  confirmed  interpretation  was  automatically  forwarded 
to  the  branch  clinic  via  a  teleprinter. 

In  addition  to  the  overread  station  in  the  Heart  Station,  there 
was  an  over read  station  in  the  Emergency  Room.  However,  the  high 
turnover  rate  in  the  ER  meant  that  there  was  low  utilization  of  the 
equipment  due  to  the  lack  of  training  of  personnel.  Originally,  the 
Coronary  Care  Unit  also  had  an  overread  station,  but  this  has  been 
removed.  ECGs  taken  in  these  locations  are  read  by  the  requesting 
physician.  ECGs  from  the  CCU  and  Emergency  Room  are  read  by  the 
cardiologists  in  the  Heart  Station.  The  only  impact  of  CAPOC  on  these 
areas  has  been  to  increase  the  capture  of  ECGs  for  reading  in  the  Heart 
Station.  Previously,  the  personnel  on  the  floors  were  relied  upon  to 
forward  the  ECGs  via  the  mail  and,  presumably,  some  were  lost  and/or 
never  sent. 

In  both  X  and  Y  data  collection  periods,  the  Heart  Station  was 
staffed  by  a  mix  of  Navy  Corpsmen  (E-3  and  E-4  levels)  and  two  to  four 
civilian  technicians.  The  civilian  technicians  took  ECGs  and  performed 
report  editing. 

The  corpsmen  were  responsible  for  all  ECGs  done  on  the  wards  and 
those  requested  when  the  Heart  Station  was  not  open.  The  corpsmen  also 
did  other  non-invasive  tests  such  as  stress  tests  and  echocardiograms. 

3.  Quantitative  Data  Collection  Methodology 


Several  different  collection  techniques  and  methods  were  utilized 
to  collect  evaluation  data  at  San  Diego  NRMC.  The  data  obtained  in¬ 
cluded:  reading  volumes  and  times  and  specialty  of  reading  staff, 
times  for  the  ECG  process,  turnaround  time,  volume  of  ECGs  done,  and 
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distribution  of  normal/abnormal  interpretations,  patient  location,  and 
types  9  of  ECGs. 

a.  Data  on  Volume  and  Type  of  ECGs 

Volume  data  were  collected  by  different  methods  in  the  X  and  Y 
Periods.  In  the  X  Period,  data  on  ECG  volumes  were  obtained  by  tallying 
the  information  from  the  log  maintained  in  the  Heart  Station.  Volume 
data  for  the  Y  Period  were  obtained  from  the  computer  logs.  These 
tallies,  coupled  with  data  obtained  during  the  data  collection  period, 
were  used  to  derive  the  breakdowns  in  volume  by  type  of  ECG. 

Information  on  the  types  of  ECGs  taken  and  read  each  morning  was  ob¬ 
tained  following  the  reading  of  the  ECGs  and  the  copying  of  the  interpre¬ 
tations;  the  observer  recorded  the  following  information:  the  log  number, 
patient  location,  where  the  ECG  was  done  (Heart  Station  or  Ward),  type  (for 
annual  physical,  pre-operative  patient,  etc.),  age,  reader,  interpretation 
(normal /abnormal) ,  whether  or  not  the  patient  had  indicated  that  a  previous 
ECG  was  available  for  a  serial  comparison,  and  if  a  prior  ECG  was  available. 
This  information  was  obtained  from  the  requisition,  the  log,  and  the  ECG  file. 

During  the  X  Period  data  collection  effort,  it  was  noted  that  while 
60  percent  of  patients  indicated  they  had  a  prior  ECG  taken  at  San  Diego, 
old  tracings  were  located  for  only  two-thirds  of  these.  Since  avail¬ 
ability  of  prior  ECGs  is  a  requirment  for  abnormal  ECGs  in  order  to  meet 
the  evaluation  criteria  for  serial  comparison,  this  finding  required 
further  investigation.  Three  reasons  for  the  lost  ECGs  were  suggested: 
the  patient  might  not  actually  have  received  the  ECG,  the  file  might 
have  been  lost,  or  the  ECG  might  have  been  misfiled  in  the  wrong  patient 
folder  or  in  the  wrong  order. 

In  order  to  investigate  these  possible  reasons  further,  80  ECGs 
were  identified  for  which  the  patient  replied  that  a  prior  ECG  had  been 
taken  at  San  Diego  but  for  which  no  old  ECG  file  was  found.  For  38 

9 

ECG  types  include:  pre-operative  ECGs,  routine  inpatient  ECGs,  STATs, 

ECGs  done  in  the  Emergency  Room  (ER) ,  routine  outpatient  ECGs,  and 
ECGs  done  as  part  of  annual  physical  examinations. 
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(48  percent)  of  these,  the  researchers  were  able  to  locate  the  old  file 
by  searching  through  the  files  nearby  the  area  where  the  file  was 
supposed  to  be  located.  Therefore,  misfiled  records  appear  to  be  a 
large  factor  in  explaining  the  number  of  prior  "lost"  ECGs. 


Reading  Time 


Reading  times  and  number  read  were  obtained  by  observing  ten  of 
the  reading  sessions  held  each  morning  in  the  Cardiology  Department. 

The  observer  recorded  the  length  of  time  and  the  number  of  physicians 
reading.  By  dividing  the  total  person-minutes  of  reading  time  by  the 
total  number  of  ECGs  read,  average  reading  times  per  ECG  were  obtained. 

An  average  reading  time  was  recorded  for  each  observation  day  and  then 
an  average  and  standard  deviation  were  calculated  for  the  entire  observa¬ 
tion  period  of  10  days. 


c .  Turnaround  Time 


In  the  X  Period,  two  different  methods  were  used  to  collect  turn¬ 
around  time^9lata.  With  the  first  method  the  log  number,  location,  and 
time  the  ECG  was  taken  were  recorded.  The  next  day  after  the  ECG  had 
been  interpreted  and  inserted  in  the  laboratory  mail  boxes,  the  boxes 
were  checked  every  half  hour  in  order  to  ascertain  if  the  interpretations 
had  been  picked  up  by  the  ward  or  clinic  personnel.  The  second  method 
was  to  attach  a  form  to  the  ECG.  The  form  contained  the  ECG  log  number 
and  the  time  the  tracing  was  taken.  The  form  contained  a  request  that 
the  nurse  or  corpsman  who  filed  the  interpretation  in  the  patient's  chart 
indicate  date  and  the  time  of  filing  and  return  the  form  to  the  Heart 
Station.  Turnaround  time  was  collected  for  ECGs  using  these  two  methods. 


turnaround  time  is  defined  as  the  time  between  the  recording  of  the  ECG 
and  the  return  of  the  written  Interpretation  to  the  appropriate  ward  or 
clinic.  It  is  usually  difficult  to  know  the  latter  time  exactly,  so 
two  different  methods  were  used  to  approximate  this  time. 
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In  the  Y  Period,  the  only  method  used  was  to  attach  forms  to  the 
ECGs.  A  total  of  1,000  forms  were  attached  to  ECGs  during  the  month  of 
October.  Of  these,  193  were  returned  with  sufficient  information  to 
determine  turnaround  time. 

4.  Quantitative  Data  Analysis 

a.  ECG  Volumes 


Monthly  ECG  volumes  for  San  Diego  NRMC  are  summarized  in  Table  3-1. 
The  data  for  the  X  Period  are  reported  for  fiscal  year  1979,  i.e., 
October  1978  to  September  1979.  Since  CAPOC  was  not  fully  operational 
in  the  first  quarter  of  fiscal  1980,  Y  Period  data  are  reported  for  the 
calendar  year  1980.  In  the  X  Period,  San  Diego  NRMC  processed  27,818 
ECGs  or  an  average  of  2,318  per  month.  These  figures  dropped  to  24,032 
and  2,002,  respectively,  in  the  Y  Period.  Because  detailed  logs  of 
patient  location  were  not  available  in  the  Y  Period,  only  two  places 
could  be  isolated  for  comparison  -  the  Coronary  Care  Unit  and  the 
Emergency  Room.  The  CCU  volume  decreased  31  percent  from  a  monthly 
average  of  313  to  220.  The  Emergency  Room  increased  from  137  ECGs  per 
month  to  164  ECGs  per  month,  an  increase  of  20  percent.  This  change 
may  be  due  to  differences  in  patient  mix  or  number  of  patients  seen. 

It  may  also  be  due  to  the  elimination  of  unnecessary  duplicate  tracings 
caused  by  poor  quality  tracings  or  lost  tracings.  In  the  Y  Period 
staff  at  San  Diego  NRMC  felt  that  CAPOC  had  caused  a  reduction  in  the 
number  of  unnecessary  repeat  tracings.  However,  no  quantitative  data 
were  available  to  evaluate  the  magnitude  of  this  impact  nor  to  estimate 
possible  overreporting  in  the  X  Period. 


64 


Arthur  D  Little.  Inc 


TABLE  3-1 

ECG  VOLUMES  FOR  SAN  DIEGO  NAVAL  REGIONAL  MEDICAL  CENTER 


X  Period 
(fiscal  year 

1979) 

Y  Period 
(calendar  year 

1980) 

October  1978 

2361 

November 

2182 

December 

1881 

January  1979 

2528 

January  1980 

2380 

February 

2365 

February 

2160 

March 

2753 

March 

1866 

April 

2860 

April 

1972 

May 

2303 

May 

1873 

June 

2380 

June 

2205 

July 

2025 

July 

2160 

August 

2058 

August 

1961 

September 

2033 

September 

1640 

October 

2173 

November 

1733 

December 

1909 

TOTAL 

27818 

TOTAL 

24032 
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Data  collected  during  the  two  data  collection  periods  and  data  from 
logs  were  used  to  estimate  breakdown  by  type  of  ECG.  The  breakdown  of 
ECG  by  types  for  the  two  periods  is  shown  in  Table  3-2. 

The  increase  in  outpatient  visits  was  reflected  in  the  increase  in 
the  percentage  of  routine  outpatient  and  annual  physical  ECGs.  Admissions 
to  the  hospital  were  down  slightly  in  1980.  The  increase  in  emergency 
room  ECGs  may  actually  be  due  to  more  accurate  reporting  by  the  CAPOC 
system.  Previously,  only  ECGs  that  were  sent  to  the  Heart  Station  by 
the  emergency  room  staff  were  entered  in  the  log  and  counted.  With 
CAPOC,  all  ECGs  done  in  the  emergency  room  were  recorded. 

b.  Service  Effectiveness  Impacts 

The  distribution  of  volume  read  by  cardiologists,  internists,  and 
other  physicians  is  shown  in  Table  3-2,  along  with  the  breakdown  of  the 
volume  according  to  normal  and  abnormal  interpretation.  These  data  have 
been  combined  with  measured  turnaround  time  in  Table  3-3,  which  summarizes 
the  resulting  service  effectiveness  for  the  X  Period,  Y  Period,  and 
projected  full  implementation  of  CAPOC. 

In  the  X  Period,  57.4  percent  of  all  ECGs  met  criteria  for  effective 
services  (acceptable  plus  preferred).  In  the  Y  Period,  this  number 
dropped  to  13.4  percent.  Two  areas  can  be  identified  as  the  cause  of 
this  drop.  Serial  comparisons  were  only  done  for  3.2  percent  of  all 
abnormal  ECGs  in  the  Y  Period,  down  from  49.2  percent  in  the  X  Period. 
Since  implementation  of  CAPOC  in  October  1979,  the  Heart  Station  had 
stopped  keeping  copies  of  ECGs.  In  addition,  all  files  for  patients 
that  have  not  had  an  ECG  within  the  last  two  years  were  discarded. 

(This  practice  was  in  existence  prior  to  CAPOC. )  These  two  events 
meant  that  ECGs  were  only  available  for  patients  whose  last  ECG  was 
in  fiscal  year  1979.  Thus,  files  were  only  pulled  for  these 
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TABLE  3-2 


SUMMARY  OF  QUANTITATIVE  DATA  FOR 
SAN  DIEGO  NAVAL  REGIONAL  MEDICAL  CENTER 


X  Period 

Volume  27,818 


Type 

Routine  Outpatient  33% 

Annual  Physical  3% 

Routine  Inpatient  42% 

Pre-Operative  15% 

STAT/ER  7% 

Reader 

Cardiologist  83.2% 

Internist  (or  Resident)  16.8% 

Other  0 

Interpretation 

Normal  41.5% 

Abnormal  58.5% 
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Y  Period 
24,032 

38.0% 

5.0% 

31.5% 

13.5% 

12.0% 


88% 

12% 

0 


36.9% 

63.1% 
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patients.  Patients  who  had  had  ECGs  since  October,  1979,  did 
not  have  copies  of  prior  tracings  because  the  automatic  storage 
and  retrieval  program  was  not  in  operation  and  Heart  Station  personnel 
did  not  do  manual  retrievals  from  the  CAPOC  system.  The  other  impact 
was  an  increase  in  measured  turnaround  time,  resulting  in  a  decrease  in 
the  percent  of  ECGs  meeting  this  criterion.  However,  since  the  X  Period 
sample  size  was  very  small,  this  change  may  not  be  significant.  Rather, 
it  may  be  du  ■  to  the  increased  number  of  ECGs  tracked  in  the  Y  Period. 

Full  implementation  could  result  in  62.1  percent  of  all  ECGs  meeting 
effective  criteria.  For  this  to  be  achieved,  several  changes  must 

occur.  The  primary  change  is  the  availability  of  automatic  storage  and 
retrieval  of  prior  ECGs.  When  this  capability  is  installed,  it  is  esti¬ 
mated  that  80  percent  of  all  abnormal  ECGs  wil1  have  a  prior  tracing 
available  for  a  serial  comparison.  Secondly,  it  would  require  that  ail 
pre-operative  ECG  reports  be  returned  to  the  wards  the  evening  before 
surgery  instead  of  the  results  being  telephoned  back.  It  would  also 
require  that  patients  sent  to  the  Heart  Station  for  an  ECG  be  given  a 
copy  of  the  unconfirmed  report  to  take  back  to  the  requesting  physician. 

This  would  result  in  all  normal  ECGs  having  a  preferred  turnaround  time. 

c. _ Sensitivity  Analysis  of  Effectiveness  Impacts 

A  sensitivity  analysis  was  performed  in  order  to  investigate  the 
possible  effects  of  errors  in  Y  Period  data  and  the  impacts  of  policy 
changes  on  service  effectiveness.  If  the  turnaround  times  in  the  Y  Period 
had  remained  unchanged  from  the  X  Period,  the  level  of  effective  ECG  service 
would  have  been  32  percent.  The  decrease  can  be  attributed  to  a  lack  of 
serial  comparisons  for  abnormals  (63  percent  of  the  volume).  If  the 
computer-generated  interpretation  for  a  normal  ECG  had  been  given  to 
the  patient  and  pre-operative  ECGs  had  been  returned  to  the  floors  in 
the  evening,  the  effectiveness  would  have  been  38  percent.  The  lack  of 
serial  comparison  is  the  key  factor  in  the  decrease  in  service  effective¬ 
ness  between  the  X  and  Y  Periods. 
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5.  Analysis  of  Qualitative  Data 

Staff  questionnaires  were  distributed  at  San  Diego  NRMC  during  both 
the  X  and  Y  Periods.  In  the  X  Period,  52  completed  questionnaires  were 
returned  by  physicians  and  18  were  returned  by  nur ses/corpsmen  and  ECG 
technicians.  In  the  Y  Period,  90  physician  questionnaires  and  17  ques¬ 
tionnaires  from  nurses/corpsmen  and  ECG  technicians  were  completed;  70 
of  the  physician  respondents  and  7  of  the  other  staff  were  on  the  staff 
of  San  Diego  NRMC  during  both  X  and  Y  Periods.  This  section  discusses 
staff  perceptions  of  ECG  services,  focusing  on  changes  that  resulted 
from  the  implementation  of  CAPOC. 

a. _ Physician  Questionnaires 

i.  ECG  Functions 

Virtually  all  physician  respondents  in  both  periods  reported  that 
they  order  ECGs  and  read  them  for  their  own  patients.  (These  staff  are 
referred  to  as  "users"  in  this  discussion.)  Approximately  40  percent 
perform  overreading  functions,  either  by  interpreting  ECGs  for  patients 
treated  by  others  or  by  checking  ECG  interpretations  done  by  others. 
Physician  respondents  reported  that  their  role  in  the  ECG  process  was 
unchanged  by  CAPOC.  Therefore,  the  respondent  base  is  assumed  to  have 
had  similar  knowledge  of  the  ECG  process  in  X  and  Y  Periods. 

ii.  _ Service  Response  Times 

Figures  3-3  through  3-7  compare  the  distribution  of  X  and  Y  Period 
responses  regarding  acceptable  service  response  times  for  STAT  and  routine 
ECG  services.  (Note  that  the  respondent  base  for  the  two  periods  was 
52  and  90,  respectively,  so  that  the  distribution  rather  than  the  number 
of  responses  is  significant.)  For  both  X  and  Y  Periods,  these  responses 
correlated  well  with  the  e-’aluation  criteria  for  turnaround  time  developed 
for  this  evaluation.  Most  of  the  respondents  believed  that  an  interpreted 
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(minutes) 


X  Period 

\y/\ STAT  ECG  in  E.R. 


Y  Period 

hd  STAT  ECG  in  E.R. 


FIGURE  3-3  ACCEPTABLE  DELAY  BETWEEN  REQUEST  FOR 

STAT  ECG  AND  TIME  IT  IS  TAKEN  IN  THE  EMER¬ 
GENCY  ROOM  -  PHYSICIAN  RESPONDENTS  TO 
STAFF  QUESTIONNAIRE  AT  SAN  DIEGO  NAVAL 
REGIONAL  MEDICAL  CENTER  -  X  AND  Y  PERIODS 


60 


X  Period 

L\\\J  STAT  ECG  in  other  areas 


Y  Period 


STAT  ECG 


in  other  areas 


(Minutes) 

FIGURE  3-4  ACCEPTABLE  DELAY  BETWEEN  REQUEST  FOR 
STAT  ECG  AND  TIME  IT  IS  TAKEN  IN  OTHER 
AREAS  -  PHYSICIAN  RESPONDENTS  TO 
STAFF  QUESTIONNAIRE  AT  SAN  DIEGO  NAVAL 
REGIONAL  MEDICAL  CENTER  -  X  AND  Y  PERIODS 
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No.  Respondents 


30 


25 


g  20 

a> 

? 

& 

M 

£  15 

d 

z 

10 


<  1 


1-4  5-8  9-24  >24 

(Hours) 


X  Period 
Y  Period 


FIGURE  3-6  ACCEPTABLE  DELAY  BETWEEN  REQUEST  FOR 

ROUTINE  ECG  AND  TIME  IT  IS  TAKEN— PHYSICIAN 
RESPONDENTS  TO  STAFF  QUESTIONNAIRE  AT 
SAN  DIEGO  NAVAL  REGIONAL  MEDICAL  CENTER  - 
X  AND  Y  PERIODS 


63 


(Hours) 


FIGURE  3-7  ACCEPTABLE  DELAY  BETWEEN  REQUEST  FOR 

ROUTINE  ECG  AND  AVAILABILITY  OF  INTERPRETATION- 
PHYSICIAN  RESPONDENTS  TO  STAFF  QUESTIONNAIRE  AT 
SAN  DIEGO  NAVAL  REGIONAL  MEDICAL  CENTER  - 
X  AND  Y  PERIODS 
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STAT  ECG  should  be  available  within  30  minutes  and  than  an  interpreted 
routine  ECG  should  be  available  within  24  hours. 


iii.  Implementation  Status 

In  the  Y  Period,  physicians  were  asked  about  the  frequency  and  cir¬ 
cumstances  of  manual  ECG  processing.  Nearly  two-thirds  were  unaware  of 
these  aspects  of  the  ECG  services.  Of  the  remainder,  most  reported 
that  manual  ECGs  are  done  almost  every  day  (2)  or  at  least  once  a  week 
(6).  The  reason  most  often  cited  was  unavailability  in  areas  where  ECGs 
are  taken  (29  responses),  CAPOC  not  available  (21)  or  not  working  (15), 
and  CAPOC  system  too  busy.  Six  respondents  felt  that  the  manual  system 
was  more  satisfactory,  citing  response  time  (especially  for  emergencies) 
(This  response  was  not  surprising  in  that  the  primary  care  physicians 
usually  do  not  have  even  the  unconfirmed,  computer-generated  interpreta¬ 
tion  when  they  see  their  patients.  In  both  periods,  these  physicians 
received  a  tracing  with  no  labeling  or  an  interpretation,  unconfirmed 
or  otherwise.  Interviews  with  staff  confirmed  that  many  are  not  even 
aware  of  CAPOC  because  interpretations  reach  the  patient  chart  or  record 
several  days  later.) 

iv.  Attributes  of  ECG  Services 


Figure  3- 8  displays  the  results  derived  from  the  portion  of  the 
staff  questionnaire  concerning  the  importance  of  certain  attributes  of 
ECG  services  and  the  degree  of  satisfaction  with  the  performance  of  the 
current  ECG  system.  In  order  to  provide  a  basis  for  comparison,  responses 
have  been  weighted  as  follows: 


Importance  Rating 
Very  important  1 
2 

3 

4 

Very  unimportant  5 


Satisfaction  Rating 
Very  satisfied  1 

2 

3 

4 

Very  dissatisfied  5 


Rating  Value 
+2 
+1 
0 

-1 

-2 
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The  sums  of  all  weighted  responses  to  each  question  are  displayed 
in  Figure  3-8  for  the  X  and  Y  Periods.  The  data  have  been  normalized 
so  that  the  maximum  possible  positive  or  negative  score  is  100. 

The  cumulative  ratings  concerning  the  importance  of  ECG  service 
attributes  are  shown  on  the  left-hand  side  of  Figure  3-8;  those  concern¬ 
ing  satisfaction  with  the  current  ECG  system  appear  on  the  right-hand 
side. 

As  can  be  seen,  importance  ratings  assigned  by  physicians  to  the 
various  attributes  were  fairly  consistent  in  the  X  and  Y  Periods.  Two 
exceptions  to  this  general  pattern  were  the  importance  of  24-hour  avail¬ 
ability  of  STAT  interpretations  and  consistency  of  ECG  readings  among 
readers.  No  reasons  related  to  CAPOC  implementation  were  identified 
to  account  for  these  differences. 

Satisfaction  ratings  increased  somewhat  for  the  response  times  for 
STAT  ECG  tracings  and  interpretations  and  24-hour  availability  of  trac¬ 
ings,  and  also  for  the  availability  of  routine  ECG  tracings.  Overall 
ratings  decreased  for  accuracy  (see  also  below)  and  completeness  of  in¬ 
terpretations  and  the  availability  of  serial  comparisons  and  experts  to 
aid  in  patient  management.  Satisfaction  with  the  technical  quality  of 
tracings  increased  by  30  percent.  Ratings  for  all  other  service  attri¬ 
butes  remained  essentially  unchanged. 

When  asked  specifically  about  changes  in  ECG  services  resulting 
from  CAPOC,  many  respondents  believed  that  most  aspects  of  ECG  services 
had  not  changed  due  to  CAPOC.  Of  those  who  believed  they  had  sufficient 
exposure  to  CAPOC  to  comment  on  changes,  many  cited  improvements  in  the 
availability  of  STAT  ECG  (39%)  and  routine  interpretations  (35%),  the 
turnaround  times  for  STAT  (42%)  and  routine  (43%)  interpretations,  and 
the  accuracy  (33%)  and  completeness  (50%)  of  interpretations.  Large 
numbers  of  those  who  commented,  cited  improvements  in  the  technical 
quality  of  tracings  (57%)  and  ease  of  reading  and  understanding  inter¬ 
pretations  (65%).  A  decrease  in  service  quality  was  reported  by  55% 
of  those  who  commented  on  the  number  of  diagnostic  and  treatment  areas 
where  ECGs  can  be  interpreted. 
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Importance 


Satisfaction  with  Current  Service 


STAT  ECGs 


STAT  ECGs 


j-51  +52 


X  Period 


[  |  V  Period 


h 


Ability  to  take  them  24  hrs.  a  day 


Number  of  hours  a  day  they  can  be  taken 


Delay  between  request  and  taking 
of  tracing  is  acceptable 


Delay  between  request  and  taking  of 
tracing 


24-hour  availability  of  interpretations 


Availability  of  interpretations 


Turnaround  time  for  interpretation 
is  acceptable 

Routine  ECGs 


+4  Turnaround  time  for  interpretation 


i-_,1  Routine  ECGs 


Ability  to  take  them  24  hours  a  day 


Number  of  hours  a  day  they  can  be  taken 


+47  +47 


Delay  between  request  and  taking 
Of  tracing  is  acceptable 


24-hour  availability  of  interpretations 


Turnaround  time  for  interpretation 
it  acceptable 


Delay  between  request  and  taking  of 
tracing 


Availability  of  interpretations 


Turnaround  time  for  interpretation 


FIGURE  3-8  CUMULATIVE  RATING  OF  ECG  SERVICE  ATTRIBUTES  - 
PHYSICIAN  RESPONDENTS  TO  STAFF  QUESTIONNAIRE 
AT  SAN  DIEGO  NRMC,  X  AND  Y  PERIODS 
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s  s 


Importance 


Satisfaction  with  Current  Service 


Other  Service  Attributes 


Other  Service  Attributes 


ECG  interpretations  are  accurate 


+74  +691 


ECG  interpretations  are  complete 
(all  aspects  noted)  +83  +79-I- 


fifl 


ECG  tracings  are  of  high  technical 
quality 


Different  readers  make  same  readings 


+52+ 


+66 


+64+ 


Interpretations  are  easy  to  read  and 
understand 


r 


+38 


>37+34 


+49 


Number  of  diagnostic  and  treatment  areas 
where  ECGs  can  be  taken 


Number  of  diagnostic  and  treatment  areas 
where  ECGs  can  be  interpreted 


Accuracy  of  ECG  interpretations 


Completeness  of  ECG  interpretations 
(all  aspects  noted) 


Technical  quality  of  ECG  tracings 


Consistency  of  interpretations  by  different 
readers 


Ease  of  reading  and  understanding  interpretations 


Prior  tracings  are  available  for  com¬ 
parison  purposes 


Exparts  are  available  to  discuss  signi¬ 
ficance  of  ECG  interpretations  in 
patient  management 


Availability  of  prior  tracings  for  comparison 
purposes 


Availability  of  experts  to  discuss  significance 
of  ECG  interpretations  in  patient  management 


FIGURE  3—8  (continued) 
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v.  Accuracy 

Table  3-4  summarizes  the  responses  of  all  users  and  readers  regard¬ 
ing  the  accuracy  of  CAPOC  interpretations  for  eight  categories  of  ECGs. 
They  generally  placed  more  confidence  in  CAPOC  readings  of  normal  than 
abnormal  ECGs.  For  abnormal  ECGs,  they  believed  CAPOC  readings  were 
more  accurate  in  identifying  ST-T  wave  abnormalities,  hypertrophy,  and 
conduction  abnormalities  than  in  detecting  rhythm  abnormalities  and 
myocardial  infarctions. 

Twenty-four  readers  and  nine  users  reported  that  computer  errors 
tended  to  be  false  positive  results;  four  readers  and  four  users  be¬ 
lieved  that  false  negative  results  were  more  likely;  and  the  remainder 
did  not  know. 

vi.  Reading  Times 

Figures  3-9  through  3-11  display  the  distribution  of  responses  re¬ 
garding  reading  times  in  the  X  and  Y  Periods.  As  indicated,  physician 
perceptions  of  their  average  reading  times  increased  for  normal  ECGs, 
decreased  for  abnormal  ECGs,  and  were  unchanged  for  serial  comparisons. 

Most  (57 %)  of  the  physicians  who  had  been  on  the  staff  before 
CAPOC  was  installed  reported  that  no  changes  had  occurred  in  the  time 
required  to  read/ interpret  ECGs  each  week.  Readers  (who  read  the 
largest  ECG  volumes)  were  more  likely  to  report  that  they  spent  less 
time  (34%  vs.  6%).  (This  is  consistent  with  the  measured  decrease  in 
reading  times.)  Physicians  spending  less  time  cited  computer  measure¬ 
ments,  legibility  of  the  CAPOC  report,  and  less  time  for  writing  the 
interpretation.  Those  who  felt  that  CAPOC  was  not  saving  time  mentioned 
most  often  the  lack  of  a  computer  interpretation  in  time  to  be  useful 
or  the  need  to  check  ECGs  anyway. 

vii.  Recalls/Referrals 

All  of  the  35  physicians  at  San  Diego  NRMC  before  CAPOC  and  after 
who  were  aware  of  the  number  of  patients  recalled  for  duplicate  tracings 
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3-9  ESTIMATED  TIME  REQUIRED  TO  READ  A  NORMAL  ECG 
PHYSICIAN  RESPONDENTS  TO  STAFF  QUESTIONNAIRE 
AT  SAN  DIEGO  NAVAL  REGIONAL  MEDICAL  CENTER  - 
X  AND  Y  PERIODS 


sec  to  min  min  min  min  min  min 
1  min 

FIGURE  3-10  ESTIMATED  TIME  REQUIRED  TO  READ  AN  ABNORMAL  ECG  - 
PHYSICIAN  RESPONDENTS  TO  STAFF  QUESTIONNAIRE 
AT  SAN  DIEGO  NAVAL  REGIONAL  MEDICAL  CENTER  - 
X  AND  Y  PERIODS 
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of  Respondents 


reported  Chat  patient  recalls  due  to  a  poor  quality  tracing  had  de¬ 
creased  since  CAPOC  (the  remainder  did  not  know).  Similarly,  they  all 
reported  that  patient  recalls  due  to  lost  tracings  had  decreased  (again 
the  remainder  did  not  know) . 

Only  19  physicians  felt  they  had  direct  knowledge  of  off-base  re¬ 
ferrals  for  cardiology  expertise  before  and  after  CAPOC.  Of  these,  12 
reported  that  referrals  were  unchanged,  five  that  referrals  had  increased, 
and  two  that  they  had  decreased.  (Between  the  X  and  Y  Periods,  visits 
to  the  Cardiology  Clinic  increased  by  3  percent.) 

viii .  Other  Comment s 

When  asked  for  an  overall  assessment  of  CAPOC,  physicians  responded 


as  follows: 

Users  Readers 

Extremely  satisfied  1  4 

Very  satisfied  9  17 

Fairly  satisfied  18  19 

Somewhat  unsatisfied  5  3 

Very  unsatisfied  -  2 

Do  not  know  9  3 


Respondents  were  also  provided  an  opportunity  to  comment  in  general 
about  ECG  services.  Several  commented  that  CAPOC  should  be  made  more 
available  in  primary  care  areas  and/or  that  there  was  insufficient  and 
outdated  ECG  equipment  in  areas  of  high  patient  concentrations.  Other 
recurring  comments  included  unavailability  of  the  unconfirmed  CAPOC  in¬ 
terpretations  and  the  delay  before  the  confirmed  interpretations  reach 
patient  charts. 

b.  Nurses  and  ECG  Technicians 

The  analysis  of  the  perceptions  of  other  staff  of  ECO  sorvic. 
before  and  after  CAPOC  has  been  focused  on  those  aspects  .  '  !'■ 
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into  which  they  have  some  unique  insight  (primarily  quality  of  tracings  and 
patients  recalled  for  repeat  tracings)  and  on  the  responses  of  the  ECG 
technicians  who  work  in  the  Heart  Station  and  thus  were  interacting 
directly  with  the  CAPOC  system. 

Five  corpsmen,  all  of  whom  worked  in  the  Heart  Station,  completed 
staff  questionnaires.  They  rated  their  overall  satisfaction  with  CAPOC 
as  follows: 


Extremely  satisfied  1 
Very  satisfied  1 
Fairly  satisfied  3 
Somewhat  unsatisfied  0 
Very  unsatisfied  0 


Specific  advantages  cited  included  automated  storage  and  retrieval 
(elimination  of  filing  of  ECGs)H  reduced  paperwork,  and  general  effi¬ 
ciency  and  quality  of  service.  Two  who  were  only  fairly  satisfied  cited 
malfunctions  of  the  portable  ECG  equipment  and  "system  flaws"  remaining 
to  be  overcome. 

The  corpsmen  reported  general  satisfaction  with  accuracy,  complete¬ 
ness,  and  consistency  of  interpretations  as  provided  by  the  CAPOC  system. 
Likewise,  they  were  very  or  fairly  satisfied  with  the  technical  quality 
of  the  tracings,  the  ease  of  reading  and  understanding  interpretations, 
and  the  availability  of  prior  tracings  for  serial  comparisons.  The  only 
corpsman  who  had  worked  at  the  Heart  Station  before  CAPOC  was  introduced 
reported  that  CAPOC  had  improved  accuracy,  consistency,  completeness, 
and  understandability  of  interpretations  and  the  turnaround  time  for 
routine  ECG  interpretations.  This  same  corpsman  reported  no  change  in 
patient  recalls  due  to  lost  ECG  reports  or  inadequate  tracings.  (The 
others  did  not  comment.) 

^This  feature  was  available  for  a  short  time,  then  removed  until  it 
could  be  adapted  to  the  patient  identification  system  within  the 
military. 
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The  12  nurse  respondents  rated  their  overall  satisfaction  with 
CAPOC  as  follows : 


Extremely  satisfied  0 
Very  satisfied  4 
Fairly  satisfied  5 
Somewhat  unsatisfied  0 
Very  unsatisfied  0 
Do  not  know  3 


Specific  advantages  cited  were  improved  service  response  times 
(although  one  nurse  felt  that  turnaround  times  for  STAT  ECGs  were  still 
too  long)  and  service  availability.  One  nurse  pointed  out  that  the 
CAPOC  interpretation  is  attached  to  the  ECG  tracing  and,  therefore, 
would  not  become  separated  or  lost.  Two  nurses  cited  system  malfunctions, 
in  one  case  particularly  the  problems  of  using  12-lead  ECG  equipment. 

The  nurses  reported  general  satisfaction  with  the  number  of  hours 
a  day  STAT  ECGs  can  be  taken  with  the  CAPOC  system,  the  technical  quality 
of  tracings,  and  the  ease  of  reading  and  interpreting  ECG  reports,  as 
provided  by  the  CAPOC  system.  Thev  were  generally  dissatisfied  with 
the  turnaround  time  for  routine  ECG  interpretations  and  the  availability 
of  serial  comparisons.  One  respondent  wrote  in  that  the  availability 
of  the  pre-operative  ECGs  was  very  unsatisfactory.  For  other  system 
attributes,  the  responses  were  mixed. 

Six  nurses  commented  on  the  impact  of  CAPOC  on  ECG  services.  There 
was  no  unanimity  of  opinion  on  any  individual  aspect  of  ECG  services. 
However,  two  believed  that  access  to  ECG  services  in  some  diagnostic  and 
treatment  areas  had  declined  and  three  believed  that  accuracy  and  com¬ 
pleteness  of  readings  had  declined.  Three  believed  that  ease  of  reading 
and  understanding  ECG  reports  had  improved,  and  two  reported  better 
technical  quality  of  ECG  tracings. 
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6.  Conclusions 


Overall  ECG  service  effectiveness  at  San  Diego  NRMC  was  57  percent 
using  manual  methods.  At  the  time  of  Y  Period  data  collection,  it  had 
decreased  to  13  percent.  Tne  decrease  was  caused  by  a  lack  of  serial 
comparisons  for  the  large  volume  of  abnormal  ECGs  read  at  the  facility. 

The  effectiveness  is  expected  to  increase  to  62  percent  when  automatic 
storage  and  retrieval  is  available  and  when  several  operational  changes 
are  instituted.  CAPOC  will  result  in  a  change  in  the  percent  of  ECGs 
meeting  preferred  service  criteria.  Physicians  at  San  Diego  NRMC  feel 
that  there  has  been  an  increase  in  the  quality  of  tracings  and  ease  of 
interpretation.  A  decrease  in  lost  ECGs  was  also  noted  as  a  benefit 
of  CAPOC. 

Twenty-three  of  29  physician  ECG  readers  surveyed  about  the  relative 
accuracy  of  manual  and  computer-assisted  interpretations  for  all  types 
of  ECGs  found  that  CAPOC  was  as  accurate  as  manual  reading;  30  of  31 
respondents  reported  that  the  accuracy  of  the  computer  equalled  or  exceeded 
the  accuracy  of  manual  reading  of  normal  ECGs.  Most  reported  that  computer 
errors  tended  to  be  false-positive  results  (calling  a  normal  ECG  abnormal). 
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1.  Introduction 


Camp  Pendleton  NRMC  consists  of  a  main  hospital,  two  small  branch 
hospitals — Twentynine  Palms  and  Barstow — and  14  clinics.  This  section 
discusses  the  ECG  services  at  the  main  hospital  and  some  information  on 
the  branch  clinics.  ECG  services  at  Twentynine  Palms  and  Barstow  are 
discussed  in  detail  elsewhere  in  this  report. 

The  physician  staff  of  the  hospital  at  Camp  Pendleton  included  one 
cardiologist  and  nine  internists.  These  staff  shared  the  responsibility 
for  reading  ECGs.  Camp  Pendleton  had  the  second  largest  volume  of  ECGs 
in  the  region.  They  accounted  for  14  percent  of  the  volume  in  1979 
(the  X  Period  evaluation)  and  16  percent  in  1980  (the  Y  Period  evaluation). 

The  185-bed  main  hospital  had  an  average  of  156  occupied  beds,  during 
this  study,  with  an  average  length  of  stay  of  5.5  days.  It  served  a  popu¬ 
lation  of  active-duty  military  personnel  numbering  22,000  and  a  large 
population  of  retired  personnel  and  dependents.  Approximately  45  percent 
of  the  inpatient  census  is  accounted  for  by  active-duty  personnel. 

The  hospital  was  opened  in  1974  and  has  modern  facilities  and 
equipment.  If  needed,  the  facility  could  have  a  maximum  capacity  of 
600  beds.  Presently,  space  not  being  used  for  Inpatient  care  has  been 
converted  to  offices,  laboratories,  and  outpatient  clinics. 

At  the  time  of  the  Y  Period  data  collection,  CAP0C  was  not  fully 
operational  at  Camp  Pendleton.  There  were  only  two  ECG  carts  that  could 
transmit  ECGs  to  the  computer.  These  carts  were  used  for  part  of  the 
ECG  volume — routine  outpatient  ECGs,  pre-operative  ECGs,  and  ECGs  as 
part  of  annual  physicals.  Transmission  problems  were  also  interfering 
with  the  use  of  the  CAP0C  system.  As  a  result  of  these  limitations, 
only  30  percent  of  the  total  ECG  volume  was  being  captured  by  CAP0C  dur¬ 
ing  the  fall  of  1980  when  Y  Period  data  were  collected. 
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The  ECG  Laboratory  at  Camp  Pendleton,  located  In  the  Internal  Medi¬ 
cine  Department,  took  approximately  one-half  of  all  ECGs  done  at  the 
hospital.  The  overall  ECG  process  has  not  been  changed  by  CAPOC. 

For  ECGs  done  in  the  ECG  Laboratory,  the  following  procedures  were 
observed  at  the  time  of  both  site  visits.  Ambulatory  patients  were  re¬ 
ferred  from  various  clinics.  They  brought  an  ECG  requisition,  which 
they  left  in  a  box  at  the  appointment  desk  within  the  Internal  Medicine 
Department.  On  a  first-come,  first-served  basis,  the  ECG  corpsman  called 
patients  from  the  waiting  room  and  took  their  ECG.  The  corpsman  then 
recorded  the  necessary  patient  information  and  retrieved  the  prior  ECG 
file  when  one  was  available.  ECGs  for  routine  ambulatory  patients  were 
nearly  all  taken  during  the  morning  on  Monday  through  Friday  at  the 
rate  of  about  five  ECGs  per  hour  with  two  ECG  corpsmen  sharing  one  three- 
channel  ECG  cart. 

For  most  non-STAT  ECGs,  the  corpsman  clipped  the  tracing  to  the 
requisition  form  and  to  the  file  folder  (containing  prior  ECGs  when 
available)  and  dropped  these  in  a  box  in  the  ECG  Laboratory.  At  the 
end  of  the  morning,  these  were  sorted  into  the  mail  boxes  of  the  cardiolo¬ 
gist  and  internists  for  reading. 

Exceptions  to  this  process  were  observed  for  some  types  of  ECGs  as 
follows : 

•  Internal  Medicine  Clinic  ECGs — the  ECG  might  be  returned 
immediately  to  the  ordering  physician  who  might  interpret 
the  ECG  before  the  patient  left  the  clinic; 

•  pre-operative  ECGs  were  sorted  separately  so  that  they 
would  be  certain  to  be  read  that  day; 

•  STAT  ECGs  were  given  to  a  reader  immediately  for  inter¬ 
pretation; 

•  pediatric  ECGs  were  sent  back  with  the  patient  to  the 
clinic.  These  might  be  held  for  Pediatric  Cardiology 
Clinic,  which  was  held  monthly,  or  they  might  be  sent 

to  San  Diego  NRMC  to  be  read  by  a  pediatric  cardiologist;  and 
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•  branch  clinic  patients  might  have  taken  the  ECG,  with 
or  without  interpretation,  back  to  the  attending  physician 
at  the  branch  clinic. 

The  corpsmen  in  the  ECG  Laboratory  took  STAT  ECGs  in  most  areas  of 
the  hospital  between  0800  and  1700  Monday  through  Friday.  The  corpsman 
usually  returned  to  the  Internal  Medicine  Department  with  the  tracing, 
brought  the  ECG  to  a  physician  for  reading,  and  then  telephoned  the  in¬ 
terpretation  back  to  the  ward.  Routine  inpatient  ECGs  on  non-ambulatory 
patients  were  usually  taken  during  the  afternoons.  Usually  two  ECG 
copies  were  made:  a  ward  copy  was  left  in  the  patient's  chart  and 
another  copy  was  brought  back  for  interpretation  and  submitted  for 
reading  via  the  mail  box  as  described  above. 

ECGs  were  also  taken  by  corpsmen  assigned  to  the  Intensive  Care 
Unit,  other  inpatient  units,  outpatient  clinics,  or  branch  clinics. 

These  ECGs  might  be  hand-delivered  or  mailed  to  the  ECG  Laboratory  for 
reading.  For  inpatients,  the  tracings  were  often  not  sent  to  the  ECG 
Laboratory  until  after  the  patient  had  been  discharged;  the  staff  in 
the  Medical  Records  Department  found  the  unread  tracings  in  the  patient's 
chart  and  mailed  them  to  the  ECG  Laboratory. 

After  the  ECGs  had  been  read,  a  photocopy  was  made  of  the  ECG  and 
interpretation  for  the  ECG  file.  The  originals  were  returned  to  the 
requesting  location.  For  wards  and  clinics  in  the  hospital  (except 
Internal  Medicine  Clinic) ,  the  ECGs  were  sorted  into  the  clinical  labora¬ 
tory  mail  boxes.  The  staff  of  each  unit  was  responsible  for  checking 
the  mail  box  for  all  laboratory  results.  Pre-operative  ECGs  were  hand- 
delivered  to  the  wards  by  the  ECG  corpsman  at  the  end  of  each  day  at 
around  1700.  Mail  to  branch  clinics  was  distributed  via  Guard  Mail, 
which  could  add  1  to  7  days  to  the  turnaround  time,  depending  upon  the 
location  of  the  clinic  and  the  mail  schedule  for  that  clinic. 

In  the  X  Period,  the  ECG  equipment  at  Camp  Pendleton  consisted  of 
three  three-channel  transmitting  carts  and  five  single-channel  carts. 
Between  the  X  and  Y  Periods,  one  of  the  three-channel  transmitting  carts 
was  sent  to  Twentynine  Palms  Marine  Corps  Base  Branch  Hospital,  leaving 
only  two  at  Camp  Pendleton.  Equipment  for  overreading  computer-generated 
ECG  interpretations  was  located  in  the  ECG  Laboratory. 
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3.  Quantitative  Data  Collection  Methodology 

Because  of  the  low  utilization  of  CAPOC  (due  to  equipment  and  trans¬ 
mission  problems)  at  Camp  Pendleton,  Y  Period  data  collection  was  limited  to 
Interviews  and  a  survey  of  staff  to  ascertain  the  status  of  ECG  services. 

X  Period  data  at  Camp  Pendleton  were  collected  principally  using 
two  techniques : 

(1)  a  data  collection  form  for  tracking  individual  ECGs 
through  the  ECG  process  and  for  collecting  data  to 
allow  comparison  with  ECG  service  criteria,  and 

(2)  log  books  for  collecting  additional  information  on 
volumes  and  types  of  ECGs. 

Ninety-two  ECGs  taken  during  a  one-week  period  were  followed  from 
the  time  they  were  taken  until  the  time  that  they  were  interpreted  and 
either  seen  by  the  attending  physician  or  were  available  for  the  physi¬ 
cian  (defined  as  the  time  the  ECG  is  returned  to  the  requesting  location). 
In  all,  91  of  the  92  ECGs  were  tracked  completely  through  this  cycle. 

In  order  to  complete  the  data  collection  forms,  the  researcher: 

(1)  observed  the  time  that  each  ECG  was  taken, 

(2)  recorded  patient  Information  and  ECG  type  from  the 
requisition  and  Log  Book, 

(3)  noted  whether  an  ECG  file  had  been  retrieved, 

(4)  observed  either  the  time  the  ECG  was  interpreted 
or  an  approximate  reading  time  (e.g.,  the  time 
that  the  interpreted  ECG  was  returned  to  the  ECG 
Laboratory) , 

(5)  noted  the  reader  and  the  interpretation  of  the  ECG, 

(6)  noted  the  time  that  the  ECG  and  interpretation 
were  mailed,  and 

(7)  called  or  visited  hospital  staff  at  the  requesting 
site  to  learn  when  the  ECG  was  returned.  This  process 
was  only  done  for  ECGs  taken  by  the  ECG  Laboratory 
corpsmen. 
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In  order  to  find  information  on  ECGs  done  elsewhere  at  Camp  Pendle¬ 
ton,  the  Log  Book  was  reviewed  for  one  month's  entries  in  order  to  learn 
how  many  ECGs  were  sent  for  reading.  During  the  one-week  period,  the 
ECGs  that  were  sent  in  for  reading  were  studied  for  turnaround  time. 

The  date  that  these  tracings  were  received  was  compared  with  the  date 
they  were  taken  in  order  to  estimate  turnaround  time. 


Quantitative  Data  Analysis 


a.  ECG  Volumes 


The  X  Period  ECG  volume  at  Camp  Pendleton  averaged  1,000  ECGs  per 
month.  Y  Period  ECG  volume  was  estimated  to  be  the  same.  The  distribu¬ 
tion  by  type  of  ECG  for  both  periods  is  given  in  Table  3-5. 


Service  Effectiveness  Impact 


The  measured  distribution  of  ECG  volume  according  to  the  medical 
specialty  of  the  reader  and  the  classification  of  the  interpretation 
(normal  or  abnormal)  are  given  in  Table  3-5.  These  findings  are  combined 
with  the  results  for  turnaround  time  in  Table  3-6,  which  summarizes  the 
levels  of  ECG  service  effectiveness  at  Camp  Pendleton  for  the  X  and  Y 
Period  and  for  full  implementation  of  CAPOC.  The  overall  level  of  ECG 
service  effectiveness  remained  at  27  percent  for  both  the  X  and  Y  Periods. 
The  level  of  preferred  services,  however,  increased  from  1  percent  to  6 
percent  of  the  ECG  volume  between  the  two  periods.  The  lack  of  change 
in  measured  service  effectiveness  can  be  explained  by  the  low  level  of 
CAPOC  utilization  at  this  site  and  the  fact  that  since  Camp  Pendleton 
had  internists  and  a  cardiologist  in  the  X  Period,  there  was  no  room  for 
improvement  in  service  effectiveness  as  measured  by  ECG  reader. 

In  order  to  achieve  maximum  benefit  from  the  CAPOC  system,  changes 
are  necessary  at  Camp  Pendleton.  Two  additional  three-channel  transmitting 


TABLE  3-5 


SUMMARY  OF  QUANTITATIVE  DATA  FOR 
CAMP  PENDLETON  NAVAL  REGIONAL  MEDICAL  CENTER 


X  Period 

Y  Period 

Volume 

12,000 

12,000 

Type. 

Routine  Outpatient 

19% 

19% 

Annual  Physical 

15% 

15% 

Routine  Inpatient 

18% 

48% 

Pre-Operative 

11% 

11% 

STAT/ER 

3% 

3% 

Pediatric 

4% 

4% 

Reader 

Cardiologist 

11% 

11.4% 

Internist 

89% 

88.6% 

Other 

0 

0 

Interpretation 

Normal 

47% 

47% 

Abnormal 

53% 

53% 
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ECG  carts  must  be  secured  for  use  in  inpatient  areas  and  the  Emergency 
Room.  Secondly,  the  automatic  storage  and  retrieval  of  ECGs  must  be 
in  operation.  A  change  in  the  ECG  process  involving  returning  routine 
normal  outpatient  and  pre-operative  ECG  tracings  and  CAPOC  interpreta¬ 
tions  with  the  patient  would  increase  the  number  of  ECGs  meeting  criteria 
for  effective  turnaround  time.  These  changes  will  result  in  an  increase 
in  ECG  service  effectiveness  to  83  percent.  The  change  will  result  from 
improved  turnaround  time  and  increased  serial  comparisons  of  abnormal 
ECGs.  Preferred  services  will  increase  to  20  percent  due  to  all  physicians 
becoming  preferred  readers  of  computer-generated  normal  ECG  interpretations. 

5.  Analysis  of  Qualitative  Data 

a.  Physician  Questionnaires 

In  the  X  Period,  46  physicians  at  Camp  Pendleton  returned  staff 
questionnaires;  in  the  Y  Period,  40  physicians  completed  questionnaires. 

Since  transmission  problems  and  lack  of  CAPOC  ECG  carts  in  primary 
care  areas  at  Camp  Pendleton  led  to  a  very  low  rate  of  ECG  capture  by 
CAPOC  at  the  time  the  Y  Period  data  were  collected,  the  analysis  of 
staff  questionnaires  from  this  9ite  is  focused  on  the  perceptions  of 
ECG  readers  (who  are  the  only  physicians  with  experience  with  CAPOC)  of 
changes  due  to  the  system.  No  attempt  is  made  to  compare  specific  re¬ 
sponses  from  the  X  and  Y  Periods  regarding  the  performance  of  the  manual 
and  automated  process  in  providing  ECG  services.  Importance  and  satis¬ 
faction  ratings  in  the  X  Period  for  various  aspects  of  ECG  services  are 
discussed  in  order  to  identify  areas  in  which  to  predict  benefits  of 
CAPOC  when  it  is  fully  operational  at  this  site. 

i.  ECG  Functions 


Ten  of  the  13  physician  ECG  readers  during  the  Y  Period  had  been 
at  Camp  Pendleton  prior  to  the  installation  of  CAPOC.  They  reported 
that  their  role  in  the  ECG  process  had  not  changed  as  a  result  of  CAPOC. 
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ii.  Status  of  Implementation 


Responses  regarding  ECG  capture  by  CAPOC  were  predictable.  Nine 
of  the  physician  readers  believed  that  ECGs  were  taken  manually  almost 
every  day  (the  other  four  did  not  know).  The  most  commonly  cited  reason 
was  CAPOC  not  available  in  areas  where  ECGs  were  taken  (nine  responses), 
followed  by  CAPOC  unavailable  (four  responses),  manual  system  more  satis¬ 
factory  (three  responses),  and  CAPOC  not  working  (two  responses). 

iii.  Service  Response  Times 


Figures  3-12  and  3-13  display  physician  responses  from  the  X 
Period  regarding  acceptable  turnaround  times  for  STAT  ECGs.  Figures 
3-14  and  3-15  provide  the  same  information  for  routine  ECGs.  These 
indicate  that  many  of  the  physicians  consider  acceptable  service  response 
times  to  be  similar  to  those  used  as  evaluation  criteria. 

iv.  Service  Attributes 


Figure  3-16  displays  responses  from  the  X  Period  regarding  the 
importance  of  various  aspects  of  ECG  services  and  the  physician's 
satisfaction  with  these  aspects  of  the  manual  ECG  process.  In  order 
to  provide  a  basis  for  comparison,  responses  have  been  weighted  as 
follows : 

Satisfaction  Rating 


Importance  Rating 
Very  important  1 
2 

3 

4 

Very  unimportant  5 


Very  satisfied 


Rating  Value 
1  +2 

2  +1 

3  0 

4  -1 

5  -2 


Very  dissatisfied 
The  sums  of  all  weighted  responses  to  each  question  are  displayed 
in  Figure  3-16.  The  maximum  possible  cumulative  rating  for  each  attri¬ 
bute  is  92  (46  physician  responses  time  +2  rating  value).  The  cumula¬ 
tive  ratings  concerning  the  importance  of  ECG  service  attributes  are 
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C2  STAT  ECG  in  E.R. 


STAT  ECG  in  other  areas 


,  Bll 


0-15  16-30  31-45  46-60  >60 

(minutes) 

FIGURE  3-12  ACCEPTABLE  DELAY  BETWEEN  REQUEST  FOR 
STAT  ECG  AND  TIME  IT  IS  TAKEN— PHYSICIAN 
RESPONDENTS  TO  STAFF  QUESTIONNAIRE  AT 
CAMP  PENDLETON  NAVAL  REGIONAL  MEDICAL 
CENTER  -  X  PERIOD 


<10  10-15  16-29  30-59  60-119  >120 
(minutes) 

FIGURE  3-13  ACCEPTABLE  DELAY  BETWEEN  REQUEST  FOR 

STAT  ECG  AND  AVAILABILITY  OF  INTERPRETATION- 
PHYSICIAN  RESPONDENTS  TO  STAFF  QUESTIONNAIRE  AT 
CAMP  PENDLETON  NAVAL  REGIONAL  MEDICAL  CENTER  - 
X  PERIOD 
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I 


FIGURE  3-14  ACCEPTABLE  DELAY  BETWEEN  REQUEST  FOR 

ROUTINE  ECG  AND  TIME  IT  IS  TAKEN— PHYSICIAN 
RESPONDENTS  TO  STAFF  QUESTIONNAIRE  AT 
CAMP  PENDLETON  NAVAL  REGIONAL  MEDICAL  CENTER  - 
X  PERIOD 


FIGURE  3-15  ACCEPTABLE  DELAY  BETWEEN  REQUEST  FOR 

ROUTINE  ECG  AND  AVAILABILITY  OF  INTERPRETATION- 
PHYSICIAN  RESPONDENTS  TO  STAFF  QUESTIONNAIRE  AT 
CAMP  PENDLETON  NAVAL  REGIONAL  MEDICAL  CENTER  - 
X  PERIOD 
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shown  on  the  left-hand  side  of  Figure  3-16;  those  concerning  satisfac¬ 
tion  with  the  manual  ECG  system  appear  on  the  right-hand  side. 

As  shown  in  Figure  3-16,  all  attributes  regarding  STAT  ECGs  re¬ 
ceived  very  high  importance  ratings.  Physicians  were  considerably  more 
satisfied  with  the  current  availability  of  tracings  for  STAT  ECGs  than 
with  the  availability  of  interpretations  for  STAT  ECGs. 

Service  attributes  regarding  availability  of  tracings  and  interpre¬ 
tations  for  routine  ECGs  all  received  relatively  low  importance  ratings. 
Physicians  were  generally  dissatisfied  with  manual  routine  ECG  services, 
in  particular  with  the  present  turnaround  time  for  interpretations. 

Other  service  attributes  rated  as  very  important  were  accuracy  and 
completeness  of  interpretations,  use  of  serial  comparison,  and  avail¬ 
ability  of  experts  to  discuss  the  significance  of  ECG  interpretations 
in  patient  management.  Physicians  were  only  marginally  satisfied  with 
these  aspects  of  the  manual  ECG  service;  they  expressed  the  most  dis¬ 
satisfaction  with  the  availability  of  serial  comparisons  and  availability 
of  experts  to  discuss  ECGs. 

In  response  to  questions  in  the  Y  Period  regarding  changes  in  ECG 
services  resulting  from  CAPOC,  at  least  50  percent  of  the  readers  sur¬ 
veyed  indicated  no  changes  had  occurred.  This  is  not  surprising,  since 
the  system  was  not  operating  very  frequently  because  of  technical  prob¬ 
lems.  Mentions  of  improvements  exceeded  mentions  of  decreases  for  com¬ 
pleteness  and  accuracy  of  interpretations  and  ease  of  reading  and  under¬ 
standing  ECG  reports.  More  responses  indicating  a  worsening  of  service 
were  obtained  regarding  availability  of  tracings  and  interpretations 
for  routine  ECGs  and  the  availability  of  prior  ECG  tracings  for  serial 
comparison. 

However,  many  of  the  service  attributes  rated  in  the  X  Period  as 
being  very  important  to  physicians  at  Camp  Pendleton  are  likely  to  im¬ 
prove  when  the  system  is  fully  implemented.  The  experience  at  San 
Diego  NRMC  suggests  that  physicians  will  perceive  improvements  in  inter¬ 
pretations  (accuracy,  consistency,  ease  of  reading  and  understanding), 
and  in  the  technical  quality  of  tracings.  If  ECG  carts  with  CAPOC  capa¬ 
bility  are  installed  in  the  Emergency  Room  and  the  inpatient  units,  im¬ 
provements  in  service  response  times  and  availability  should  also  be  achieved. 

97 


Arthur  D  Little  Inc 


Importance 

STAT  ECO* 


Ability  to  take  them  24  hrt.  a  day 


Delay  between  request  and  taking 
of  tracing  is  acceptable 


24-hour  availability  of  interpretations 


Turnaround  time  for  interpretation 
is  acceptable 

Routine  ECGs 


Ability  to  take  them  24  hours  a  day 


Delay  between  request  and  taking 
of  tracing  is  acceptable 


24-hour  availability  of  interpretations 


Turnaround  time  for  interpretation 
is  acceptable 


i 


Satisfaction  with  Currant  Service 

STAT  ECGs 


Number  of  hours  a  day  they  can  be  taken 


Delay  between  request  and  taking  of 
tracing 


Availability  of  interpretations 


Turnaround  time  for  interpretation 


Routine  ECGt 


Number  of  hours  a  day  they  can  be  taken 


Delay  between  request  and  taking  of 
tracing 


Availability  of  interpretations 


Turnaround  time  for  interpretation 


FIGURE  3-16  CUMULATIVE  RATING  OF  ECG  SERVICE  ATTRIBUTES  - 
PHYSICIAN  RESPONDENTS  TO  STAFF  QUESTIONNAIRE 
AT  CAMP  PENDLETON  NAVAL  REGIONAL  MEDICAL 
CENTER  -  X  PERIOD 
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Importance 


Satisfaction  with  Currant  Sarvice 


Other  Service  Attributes 


Other  Service  Attributes 


+41  Kg’ 


Ability  to  take  ECGi  in  all  diagnostic 
and  treatment  areas 


Number  of  diagnostic  and  treatment  areas 
where  ECGs  can  be  taken 


+  10J1 

Aoility  to  interpret  ECGs  in  all  diagnostic  ] 

and  treatment  areas  “ 


ba+5 


Number  of  diagnostic  and  treatment  areas 
where  ECGs  can  be  interpreted 


ECG  interpretations  are  accurate 


Wim  +29 


Accuracy  of  ECG  interpretations 


ECG  interpretations  are  complete 
(all  aspects  noted) 


Completeness  of  ECG  interpretations 
(all  aspects  noted) 


mi 


ECG  tracings  are  of  high  technical 
quality 


Technical  quality  of  ECG  tracings 


+55  m 


Different  readers  make  same  readings 


Consistency  of  interpretations  by  different 


+74  f§ 


interpretations  are  aaey  to  read  and 


Eaea  of  reading  and  understanding  interpretations 


Prior  tracings  are  available  for  com¬ 
parison  purposes 


Availability  of  prior  tracings  for  comparison 
purposes 


Experts  are  available  to  discuss  signi¬ 
ficance  of  ECG  interpretations  in 
patient  management 


-5  Availability  of  experts  to  discuss  significance 
of  ECG  interpretations  in  patient  management 


FIGURE  3-1®  (Continued) 
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Responses  from  physician  readers  in  the  Y  Period  regarding  the 
accuracy  of  CAPOC  vs.  manual  ECG  interpretation  indicated  that  CAPOC 
was  viewed  as  less  reliable  for  abnormal  ECGs,  particularly  myocardial 
infarctions,  hypertrophy,  and  conduction  abnormalities.  For  normal 
ECGs  and  other  types  of  abnormal  ECGs,  no  clear  pattern  emerged,  since 
roughly  equal  numbers  of  physicians  believed  that  CAPOC  was  more  accu¬ 
rate,  that  manual  reading  was  more  accurate,  or  that  there  was  no  dif¬ 
ference  between  the  two  modes.  Four  readers  reported  that  computer 
errors  tend  to  be  false/negative  results;  one  believed  that  false/ 
positive  results  are  more  likely  (five  did  not  know). 

vi.  Reading  Times 

Responses  from  the  X  Period  regarding  time  required  to  read  ECGs 
are  displayed  in  Figures  3-17  through  3-19. 

In  the  Y  Period,  most  readers  reported  no  change  in  the  time  re¬ 
quired  to  read/interpret  ECGs.  Again,  comments  were  made  to  the  effect 
that  the  system  had  not  been  fully  implemented. 

vii.  Recalls/Referrals 


Most  physician  readers  reported  that  patient  recalls  due  to  poor 
quality  tracings  or  lost  ECGs  were  not  changed  by  CAPOC  as  currently 
implemented.  (Two  respondents  believed,  however,  that  tracings  were 
lost  more  frequently.)  Generally,  off-base  referrals  were  also  believed 
to  be  unchanged  by  the  introduction  of  CAPOC. 
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No.  of  Respondents 


FIGURE  3-17  ESTIMATED  TIME  REQUIRED  TO  READ  A  NORMAL  ECO  - 
PHYSICIAN  RESPONDENTS  TO  STAFF  QUESTIONNAIRE 
AT  CAMP  PENDLETON  NAVAL  REGIONAL  MEDICAL  CENTER  - 
X  PERIOD 


1  min 


FIGURE  3-18  ESTIMATED  TIME  REQUIRED  TO  READ  AN  ABNORMAL  ECG  - 
PHYSICIAN  RESPONDENTS  TO  STAFF  QUESTIONNAIRE 
AT  CAMP  PENDLETON  NAVAL  REGIONAL  MEDICAL  CENTER  - 
X  PERIOD 
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viii.  Other  Comments 


Physicians  reported  overall  satisfaction  with  CAPOC  as  follows: 

Readers 


Extremely  satisfied  0 
Very  satisfied  1 
Fairly  satisfied  1 
Somewhat  unsatisfied  4 
Very  unsatisfied  2 
Do  not  know  4 


Recurring  comments  about  ECG  services  included  lack  of  access  to 
routine  ECGs,  equipment  allocation  (especially  the  lack  of  CAPOC  capa¬ 
bility  in  the  Emergency  Room,  and  the  lack  of  12-lead  machines  in  patient 
care  areas)  and  malfunction. 

6.  Conclusions 

At  the  time  of  the  Y  Period  evaluation,  only  30  percent  of  ECGs 
done  at  Camp  Pendleton  were  being  processed  by  CAPOC  due  to  transmission 
problems  and  lack  of  carts  in  the  inpatient  wards  and  Emergency  Room. 
Overall  service  effectiveness  did  not  change  from  the  baseline  of  27 
percent  with  CAPOC  implementation;  however,  the  preferred  service  level 
did  increase  to  6  percent  from  1  percent  in  the  X  Period.  When  new 
carts  are  obtained,  transmission  problems  solved,  turnaround  time 
improved,  and  automatic  storage  and  retrieval  are  Implemented,  the 
service  effectiveness  will  Increase  significantly  to  83  percent. 
Physicians  were  generally  dissatisfied  with  CAPOC  services  available 
in  the  fall  of  1980  due  to  unresolved  transmission  problems  and  lack 
of  equipment,  which  prevented  the  system  from  being  used  effectively. 


103 


Arthur  D  Little  Inc 


C.  USAF  HOSPITAL,  NELLIS  AIR  FORCE  BASE 


v.v 


* 


1. 


Introduction 


The  USAF  Hospital  at  Nellis  is  the  only  Air  Force  facility  included 
among  the  primary  evaluation  sites.  Site  visits  to  the  secondary  sites 
demonstrated  that  Nellis  is  quite  typical  of  small  Air  Force  hospitals, 
at  least  insofar  as  ECG  services  are  concerned.  Nellis,  located  on  the 
outskirts  of  Las  Vegas,  Nevada,  has  a  40-bed  hospital,  which  serves  active- 
duty  personnel  from  this  base  and  their  dependents  and  a  growing  popula¬ 
tion  of  retired  military  personnel.  Inpatient  admissions  were  2,400  in 
the  X  Period  and  2,600  in  the  Y  Period.  Outpatient  visits  to  clinics 
declined  from  200,000  in  the  X  Period  to  188,000  in  the  Y  Period.  The 
staff  included  19  physicians,  three  of  whom  were  internists,  in  both 
periods. 

The  hospital  contains  a  two-bed  Close  Observation  Unit,  which  is 
located  on  the  ward  across  from  the  nursing  station  in  order  to  facili¬ 
tate  visual  monitoring  of  patients.  For  difficult  diagnostic  or  treat¬ 
ment  problems,  patients  are  most  often  referred  to  nearby  civilian  hos¬ 
pitals. 


v 

V 


>  I 

N  ] 


2.  ECG  Service  Organization  and  Staffing 


At  Nellis,  ECGs  were  taken  and  processed  in  four  locations:  the 
Internal  Medicine  Clinic,  the  Flight  Examination  Clinic,  the  Emergency 
Room,  and  the  wards.  The  ECG  process  at  the  time  of  the  X  Period  data 
collection  is  described  below.  Changes  that  have  resulted  from  CAPOC 
are  also  summarized. 

In  the  Internal  Medicine  Clinic,  most  patients  had  a  scheduled 
appointment  for  an  ECG.  Unscheduled  ECGs  were  taken  if  time  permitted. 
Patients  arrived  at  the  Clinic  with  a  requisition.  One  of  two  Interna] 
Medicine  technicians  took  the  ECG  and  recorded  patient  information  in 
the  Log  Book.  Then  the  tracing  was  left  for  mounting,  which  was  usually 
done  in  the  afternoon.  (The  mounting  will  be  eliminated  by  CAPOC.) 
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An  internist  read  the  ECGs  the  next  morning.  Prior  tracings  were 
not  usually  retrieved  for  serial  comparison.  There  was  no  three-channel 
transmitting  cart  in  the  Internal  Medicine  Clinic  during  the  Y  Period 
data  collection.  Thus,  the  ECG  process  had  not  changed  and  the  ECGs 
were  not  impacted  by  CAPOC.  There  is  a  three-channel  transmitting  cart 
on  order  for  Nellis,  which  will  be  installed  in  the  Internal  Medicine 
Clinic. 

The  Flight  Examination  Clinic  took  ECGs  as  part  of  annual  flight 
examinations.  Corpsmen  assigned  to  the  Clinic  took  the  tracings.  After 
all  laboratory  tests  had  been  completed,  the  flight  surgeon  examined  the 
patient,  usually  the  same  day.  The  ECG  was  often  read  during  the  exami¬ 
nation.  One  of  the  flight  surgeons  was  an  internist  and  read  most  of 
these  ECGs.  The  interpretation  was  stored  in  the  patient's  record  in 
the  Clinic.  This  record  was  always  available  to  the  flight  surgeon  for 
obtaining  prior  ECGs  for  serial  comparison.  A  copy  of  the  ECG  was  also 
forwarded  to  Brooke  Air  Force  Base  in  Texas,  where  a  central  file  of 
flight  ECGs  was  kept.  The  ECG  was  also  interpreted  at  Brooke  and  Nellis 
was  contacted  if  follow-up  examinations  were  recommended  for  the  patient. 
All  ECGs  in  the  Flight  Examination  Clinic  were  being  processed  by  CAPOC. 
The  unconfirmed  interpretation  was  attached  to  the  ECG  tracing  to  be 
seen  by  the  examining  physician. 

ECGs  taken  in  the  Emergency  Room  were  usually  taken  by  corpsmen 
assigned  to  this  service.  They  were  read  immediately  by  the  General 
Medical  Officer  (who  was  most  often  a  general  practitioner,  but  some¬ 
times  was  an  internist  since  all  physicians  shared  this  duty).  If  the 
physician  needed  to  consult  another  physician,  an  internist  was  usually 
available  during  the  day,  but  had  to  be  called  to  come  in  at  night  and 
on  weekends.  The  patient's  medical  record  was  retrieved  when  possible. 
ECGs  taken  in  the  Emergency  Room  were  processed  by  CAPOC.  The  unconfirmed 
interpretation  was  available  for  the  physician. 

On  the  inpatient  wards,  ECGs  were  usually  taken  by  corpsmen  and 
nurses  assigned  to  these  units.  The  patients  included  those  in  the 
Close  Observation  Unit,  non-ambulatory  patients,  and  those  who  needed 
tracings  at  night  and  on  weekends.  These  ECGs  were  read  by  an  internist 
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who  was  usually  caring  for  most  such  patients  on  these  wards.  No  in¬ 
patient  ECGs  were  being  processed  by  CAPOC  during  the  Y  Period  data 
collection.  The  cart,  which  is  on  order,  is  expected  to  be  used  for 
these  ECGs,  in  addition  to  the  ones  taken  in  the  Internal  Medicine  Clinic. 

In  conclusion,  only  54  percent  of  all  ECGs  taken  at  Nellis  were 
being  processed  by  CAPOC.  When  the  new  cart  arrives,  all  ECGs  are  ex¬ 
pected  to  be  processed  by  CAPOC.  Nellis  receives  only  unconfirmed  in¬ 
terpretations.  There  is  no  provision  for  overreading  of  these  ECGs  by 
a  cardiologist  at  another  site,  such  as  San  Diego  NRMC. 

3.  Quantitative  Data  Collection  Methodology 

Several  different  methods  were  employed  to  collect  X  Period  data 
at  Nellis,  including  observation  of  the  ECG  process,  review  of  existing 
records  and  log  books,  completion  of  data  collection  forms,  and  inter¬ 
views  with  staff  in  all  areas  of  the  hospital  where  ECGs  were  taken. 

The  data  collection  methods  for  the  Y  Period  included  interviews  with 

pertinent  staff  (i.e. ,  physicians  and  corpsmen),  review  of  the  ECG  pro¬ 
cess,  and  review  of  ECG  Log  Books-  The  data  obtained  included  ECG 
volumes  by  type  of  ECG,  specialty  of  reading  staff,  turnaround  times, 
and  use  of  serial  comparison. 

4.  Quantitative  Data  Analysis 

a.  Data  on  Volume  and  Type  of  ECG 

The  annual  ECG  volume  in  the  X  Period  was  3,900.  This  number  de¬ 
creased  by  12  percent  to  3,147  in  the  Y  Period.  The  medical  staff  at¬ 
tributed  the  decrease  to  a  change  in  the  criteria  for  ordering  an  ECG 
as  part  of  an  annual  physical  examination.  The  decrease  might  also  be 
due  to  the  6  percent  decrease  in  outpatient  visits  between  the  two 
periods. 
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Service  Effectiveness  Impact 


Changes  in  the  types  of  ECGs,  the  volumes  read  by  different  readers 
and  the  category  of  interpretation  are  summarized  in  Table  3-7.  Table 
3-8  shows  the  levels  of  ECG  service  effectiveness  at  Nellis  for  the  X 
and  Y  Periods  and  for  full  implementation  of  CAPOC.  Service  effective¬ 
ness  increased  from  41  percent  in  the  X  Period  to  59  percent  in  the  Y 
Period.  There  were  two  areas  that  contributed  to  the  improvement: 
improved  turnaround  time  and  all  physicians  becoming  preferred  readers 
for  normal  ECGs  with  the  back-up  of  the  computer.  When  the  new  cart 
is  available  and  all  ECGs  are  captured  by  CAPOC,  the  level  of  service 
effectiveness  is  expected  to  be  84  percent.  Since  Nellis  is  not 
connected  with  a  CAPOC  overreading  station,  this  site  does  not  benefit 
from  the  ability  to  have  abnormal  ECGs  read  by  a  cardiologist  off-site, 
or  to  have  prior  tracings  automatically  retrieved  for  serial  comparison. 

5.  Analysis  of  Qualitative  Data 


In  the  X  Period,  staff  questionnaires  were  returned  by  11  physicians 
at  USAF  Hospital,  Nellis  Air  Force  Base.  Ten  physicians  returned  Y 
Period  questionnaires;  six  of  the  respondents  performed  ECG  reading 
functions.  One  CAPOC  ECG  cart  is  utilized  in  both  the  Flight  Surgeon's 
Office,  and  the  adjacent  Emergency  Room.  Therefore,  at  the  time  of  the 
Y  Period  data  collection,  ECG  capture  by  CAPOC  was  limited  for  the  most 
part  to  STAT  ECGs  in  the  Emergency  Room  and  annual  physical  ECGs  done 
in  the  Flight  Surgeon's  Office.  Accordingly,  the  following  analysis 
of  questionnaire  results  has  been  focused  on  physician  perceptions  of 
CAPOC  impacts  on  these  aspects  of  ECG  services.  Furthermore,  because 
of  the  small  volume  of  ECGs  processed  at  this  site  and  the  small  number 
of  physicians  on  the  staff,  qualitative  and  descriptive  analysis  was 
undertaken  rather  than  data  tabulations  of  the  type  done  for  San  Diego 
NRMC. 

In  the  Y  Period,  eight  of  the  ten  physician  respondents  at  Nellis 
had  been  on  the  staff  prior  to  the  implementation  of  CAPOC.  They  re¬ 
ported  that  CAPOC  had  not  altered  their  role  in  the  ECG  process. 
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TABLE  3-7 

SUMMARY  OF  QUANTITATIVE  DATA  FOR 
USAF  HOSPITAL,  NELLIS  AIR  FORCE  BASE 
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Regarding  ECG  capture  by  CAPOC,  respondents  indicated  that  manual 
ECGs  were  done  almost  every  day  (four  responses)  or  at  least  once  a 
week  (three  responses).  Reasons  given  included  CAPOC  not  working  (six 
responses),  CAPOC  system  too  busy  (four),  CAPOC  not  available  in  all 
areas  where  ECGs  were  taken  (three),  and  manual  system  more  satisfactory 
(two).  With  regard  to  the  last  reason,  one  physician  believed  that 
longer  ECG  tracings  (produced  by  the  single-channel  carts)  were  easier 
to  read . 

Figure  3-20  displays  responses  from  the  X  Period  survey  concerning 
the  relative  importance  of  various  attributes  of  ECG  services  and  the 
degree  of  satisfaction  with  the  ECG  services  provided  in  the  manual 
mode.  In  order  to  provide  a  basis  for  comparison,  responses  have  been 
weighted  as  follows: 


Importance  Rating 
Very  important  1 

2 

3 

4 

Very  unimportant  5 


Satisfaction  Rating 
Very  satisfied  1 

2 

3 

4 

Very  dissatisfied  5 


Rating  Value 
+2 
+1 
0 

-1 

-2 


The  sums  of  all  weighted  responses  to  each  question  are  displayed 
in  Figure  3-20.  The  maximum  possible  cumulative  rating  for  each  attri¬ 
bute  is  22  (11  physician  responses  times  +2  rating  value) .  The  cumula¬ 
tive  ratings  concerning  the  importance  of  ECG  service  attributes  are 
shown  on  the  left-hand  side  of  Figure  3-20;  those  concerning  satisfac¬ 
tion  with  the  current  ECG  system  appear  on  the  right-hand  side. 

System  attributes  regarding  STAT  ECGs  all  were  viewed  as  important, 
especially  the  24-hour  availability  of  tracings,  which  all  11  physicians 
rated  as  very  important.  Physicians  were  quite  satisfied  with  the 
availability  of  tracings  for  STAT  ECGs,  but  generally  dissatisfied  with 
the  availability  of  interpretations  under  the  current  ECG  service. 
Relative  importance  ratings  for  attributes  of  routine  ECGs  were  low. 
Physicians  at  Nellis  appeared  to  be  only  somewhat  satisfied  with  current 
routine  ECG  services,  but  least  satisfied  with  the  availability  of  in¬ 
terpretations  for  routine  ECGs. 
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Other  system  attributes  rated  as  very  important  were  accuracy  of 
interpretations,  high  quality  of  ECG  tracings,  ease  of  reading  and  under¬ 
standing  interpretations,  and  availability  of  prior  tracings  for  compari¬ 
son  purposes.  Physicians  were  quite  satisfied  with  the  accuracy  and 
completeness  of  interpretations  provided  by  the  manual  ECG  service  and 
marginally  satisfied  with  the  number  of  diagnostic  and  treatment  areas 
where  ECGs  could  be  taken  and  interpreted,  the  consistency  and  ease  of 
reading  and  understanding  interpretations,  and  the  availability  of  serial 
comparisons.  Their  responses  indicated  that  they  were  somewhat  dissatis¬ 
fied  with  the  availability  of  experts  to  discuss  the  significance  of 
ECG  interpretations  in  patient  management. 

In  response  to  a  series  of  questions  in  the  Y  Period  survey  regard¬ 
ing  CAPOC-induced  changes  in  various  aspects  of  ECG  services,  physicians 
confirmed  the  improvement  in  STAT  ECG  services  (due  to  availability  of 
CAPOC  service  in  the  Emergency  Room  at  night);  availability  of  and  turn¬ 
around  times  for  STAT  ECG  tracings  and  interpretations  were  reported  to 
be  improved  by  six  respondents  (the  remainder  indicated  they  did  not 
know).  Improvements  were  noted  in  accuracy  (six  vs.  none)  and  complete¬ 
ness  (five  vs.  none)  of  interpretations,  ease  of  reading  and  understand¬ 
ing  ECG  reports  (five  vs.  one),  and  quality  of  tracings  (six  vs.  none). 

Physicians  were  asked  to  compare  the  accuracy  of  CAPOC  and  manual 
ECG  interpretations  for  normal  and  five  types  of  abnormal  ECGs.  For 
every  category,  these  physicians  believe  that  manual  reading  was  more 
accurate.  They  believed  that  computer  errors  tend  to  be  false/positive 
results  (six  responses)  rather  than  false/negative  results  (one  response). 

Figures  3-21  through  3-23  summarize  responses  from  the  X  Period 
regarding  time  required  to  read  ECGs.  In  the  Y  Period,  most  (six)  physi¬ 
cians  noted  no  change  as  a  result  of  CAPOC;  one  each  noted  an  increase 
or  decrease  of  time  devoted  to  these  functions.  The  one  who  believed 
CAPOC  required  more  time  questioned  the  ability  of  CAPOC  to  identify 
abnormals.  Those  indicating  no  change  noted  the  need  to  check  computer 
results.  Weekly  ECG  volumes  for  respondents  from  Nellis  averaged  11 
and  10  ECGs  for  the  X  and  Y  Periods,  respectively. 
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Importance 


Satisfaction  with  Currant  Service 


ST  AT  ECO* 


+22 


Ability  to  taka  them  24  hrs.  a  day 

+19 


Delay  between  request  and  taking 
of  tracing  is  acceptable 

+18 


24-hour  availability  of  interpretations 


+16 


Turnaround  time  for  interpretation 
is  acceptable 

Routine  ECGa 


-7 

Ability  to  take  them  24  hours  a  day 


+9 

Delay  between  request  and  taking 
of  tracing  is  acceptable 


24-hour  availability  of  interpretations  -10 


+8 


Turnaround  time  for  interpretation 
is  acceptable 


STAT  ECGs 


Number  of  hours  a  day  they  can  be  taken 


Delay  between  request  and  taking  of 
tracing 


Availability  of  interpretations 


Turnaround  time  for  interpretation 


Routine  ECGa 


Number  of  hours  a  day  they  can  be  taken 


Delay  between  request  and  taking  of 
tracing 


Availability  of  interpretations 


Turnaround  time  for  interpretation 
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FIGURE  3-20  CUMULATIVE  RATING  OF  ECG  SERVICE  ATTRIBUTES  -  PHYSICIAN 
RESPONDENTS  TO  STAFF  QUESTIONNAIRE  AT  USAF  HOSPITAL 
NELLIS  AIR  FORCE  BASE  -  X  PERIOD 
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Importance 

Other  Service  Attribute* 


Satisfaction  with  Currant  Service 


Other  Service  Attributee 


+  1 

Ability  to  teke  ECGt  in  ell  diagnostic 
end  treatment  area* 


-1 

Ability  to  interpret  ECG*  in  all  diagnostic 
and  treatment  areas 


+20 


ECG  interpretations  are  accurate 


+13 


ECG  interpretations  are  complete 
(all  aspects  noted) 

+16 


ECG  tracings  are  of  high  technical 
quality 


+12 


Different  readers  make  same  readings 
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Intarpratations  are  easy  to  raad  and 
undarstand 
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Prior  tracings  ara  available  for  com¬ 
parison  purposes 
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Experts  are  available  to  discuss  signi¬ 
ficance  of  ECG  interpretations  In 
patient  management 


Number  of  diagnostic  and  treatment  areas 
where  ECGt  can  be  taken 


Number  of  diagnostic  and  treatment  areas 
where  ECGt  can  be  interpreted 


Accuracy  of  ECG  interpretations 


Completeness  of  ECG  interpretations 
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Availability  of  prior  tracings  for  comparison 
purposes 


Availability  of  experts  to  discuss  significance 
of  ECG  interpretations  in  patient  management 


FIGURE  3-20  (continued) 
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No.  of  Respondents 


FIGURE  3-21  ESTIMATED  TIME  REQUIRED  TO  READ  A  NORMAL  ECG  - 
PHYSICIAN  RESPONDENTS  TO  STAFF  QUESTIONNAIRE 
AT  USAF  HOSPITAL,  NELLIS  AIR  FORCE  BASE  -  X  PERIOD 


1  min 


FIGURE  3-22  ESTIMATED  TIME  REQUIRED  TO  READ  AN  ABNORMAL  ECG  - 
PHYSICIAN  RESPONDENTS  TO  STAFF  QUESTIONNAIRE 
AT  USAF  HOSPITAL,  NELLIS  AIR  FORCE  BASE  -  X  PERIOD 
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•  VI 
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FIGURE  3—23  ESTIMATED  TIME  REQUIRED  TO  MAKE  A  SERIAL  COMPARISON  WITH 
A  PRIOR  ECG  TRACING  -  PHYSICIAN  RESPONDENTS  TO  STAFF 
QUESTIONNAIRE  AT  USAF  HOSPITAL,  NELLIS  AIR  FORCE  BASE  - 
X  PERIOD 
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In  the  Y  Period,  physicians  did  not  believe  the  number  of  off-base 
referrals  had  been  changed  by  CAPOC.  Two  believed  that  patient  recalls 
due  to  poor  quality  tracings  had  increased  (one  noted  a  decrease)  and 
one  respondent  reported  that  recalls  due  to  loss  of  ECGs  had  declined. 

Physicians  rated  their  overall  satisfaction  with  CAPOC  as  follows: 


Extremely  satisfied  0 
Very  satisfied  3 
Fairly  satisfied  6 
Somewhat  unsatisfied  1 
Very  unsatisfied  0 


Those  who  are  generally  satisfied  noted  quick  reporting  but  some 
questioned  the  accuracy  of  interpretations.  One  respondent  noted  system 
downtime  as  a  problem.  Several  comments  indicated  that  interpretations 
for  routine  ECGs  were  not  always  received  by  the  ordering  physician. 

6.  Conclusions 


The  overall  ECG  service  effectiveness  at  Nellis  increased  from  41 
percent  with  manual  services  to  59  percent  in  the  Y  Period.  This  increase 
occurred  despite  only  54  percent  of  ECGs  being  processed  by  CAPOC  due 
to  a  lack  of  a  cart  for  certain  areas.  When  the  cart  becomes  available, 
the  service  effectiveness  will  increase  to  84  percent.  Physician  staff 
commented  on  the  rapid  availability  of  an  unconfirmed  interpretation  as 
a  benefit  of  CAPOC. 
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D.  TWENTYNINE  PALMS  MARINE  CORPS  BASE  BRANCH  HOSPITAL 


1.  Introduction 


Twentynine  Palms  Branch  Hospital  is  the  smallest  inpatient  treatment 
facility  included  among  the  primary  evaluation  sites.  Administratively, 
it  is  a  component  of  the  Command  of  Camp  Pendleton  Naval  Regional  Medical 
Center.  However,  for  ECG  services,  Twentynine  Palms  operates  quite 
autonomously  from  Camp  Pendleton.  The  average  monthly  volume  was  100 
ECGs  in  1979,  the  X  Period. 

Twentynine  Palms  is  located  in  a  remote  desert  area,  approximately 
a  2-hour  drive  from  the  nearest  referral  hospital.  When  referrals  are 
indicated,  patients  may  be  transported  to  either  Camp  Pendleton,  San 
Diego,  or  a  civilian  hospital  in  Loma  Linda,  California. 

The  hospital  had  a  maximum  capacity  of  34  beds  with  an  average 
occupancy  of  11  beds  in  1979.  Reportedly,  occupancy  varied  from  0  to 
34  beds.  The  hospital  primarily  served  the  20,000  active-duty  military 
personnel  assigned  to  this  Marine  Corps  Base,  although  this  number  in¬ 
creased  substantially  at  times  when  major  military  maneuvers  were  held 
on  base.  The  population  of  retired  military  personnel  treated  at  Twenty- 
nine  Palms  was  small,  but  had  been  increasing  in  recent  years  due  to  the 
proximity  of  the  Palm  Springs  resort  area. 

The  hospital  staff  in  the  X  Period  included  8  physicians:  6  in 
general  medicine,  1  pediatrician,  and  1  specialist  in  obstetrics  and 
gynecology.  CAP0C  had  not  been  implemented  at  Twentynine  Palms  during 
the  Y  Period  data  collection  because  of  unresolved  transmission  problems. 


2.  ECG  Service  Organization  Staffing 

There  was  no  structured  ECG  service,  per  se,  at  Twentynine  Palms. 

Any  corpsman  caring  for  a  patient — whether  in  the  Emergency  Room,  Physical 
Examination  Clinic  or  on  the  Military  (Inpatient)  Ward — might  take  an 
ECG  when  indicated.  Three  one-channel  carts  were  available  for  taking 
ECGs  in  these  locations.  At  the  time  of  baseline  data  collection, 
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approximately  10  corpsmen  took  most  of  the  ECGs,  but  reportedly  any  of 
the  100  staff  could  potentially  take  an  ECG. 

ECGs  were  read  by  the  attending  and  ordering  physician  often  imme¬ 
diately  after  they  were  taken,  but  always  within  one  day.  One  physician, 
who  had  a  particular  interest  in  ECGs,  served  as  overreader  for  the 
tracings  when  requested  by  the  attending  physician.  The  interpretation 
was  usually  written  in  the  patient's  chart  or  on  the  ECG  itself,  which 
was  then  inserted  into  the  patient's  chart.  The  chart  was  then  available 
so  that  if  a  prior  ECG  was  contained  in  the  record,  a  serial  comparison 
could  be  made. 

Twentynine  Palms  is  expected  to  have  two  three-channel  transmitting 
ECG  carts  for  use  with  CAPOC.  One  will  be  in  the  Family  Practice  Clinic 
adjacent  to  the  Emergency  Room  and  the  other  will  be  for  use  in  the  in¬ 
patient  wards.  The  CAPOC  printer  will  be  located  in  the  Family  Practice 
Clinic. 

3.  Quantitative  Data  Collection  Methodology 

Quantitative  data  on  ECG  services  at  Twentynine  Palms  was  difficult 
to  obtain  because  of  the  low  ECG  volume.  Therefore,  data  collection  was 
designed  specifically  to  capture  the  most  relevant  information  for  com¬ 
parison  with  the  ECG  service  criteria. 

a.  Data  on  Volumes  and  Types  of  ECG 

Information  on  volumes  was  obtained  by  reviewing  the  ECG  Log  and 
the  monthly  report  that  is  prepared  for  accounting  purposes.  Since  these 
numbers  differed  from  each  other,  interviews  with  staff  were  conducted 
in  order  to  resolve  conflicts.  An  estimate  of  actual  volume  and  types 
of  ECGs  was  made,  based  upon  these  interviews. 

b.  Turnaround  Time 


Data  on  turnaround  time  were  obtained  by  asking  the  staff  to  complete 
a  form  specially  designed  for  this  purpose.  These  forms  were  attached 
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to  the  ECG  after  it  was  taken.  The  reader  (and  attending  physician)  was 
asked  to  fill  in  the  time  the  ECG  was  interpreted  and  the  time  that  the 
tracing  and  interpretation  were  returned  to  the  patient’s  medical  record. 
On  this  form,  other  information  was  requested  concerning  the  type  of 
ECG,  the  patient’s  age,  and  where  the  ECG  was  taken.  A  total  of  16  forms 
were  fully  completed  and  four  others  were  partially  completed. 

Other  information  on  the  staff  reading,  serial  comparison,  interpre¬ 
tation  (normal  or  abnormal),  and  turnaround  time  was  obtained  through 
staff  interviews.  No  data  collection  was  done  in  the  Y  Period  since 
the  CAPOC  system  was  not  being  used. 


Juantitative  Data  Analysis 


a.  ECG  Volumes 

A  review  of  the  ECG  Log  for  the  months  between  September,  1978,  and 
September,  1979,  and  the  monthly  morbidity  reports  prepared  by  the  hospi¬ 
tal  for  January,  1979  ,  through  July,  1979,  showed  differences  in  reported 
volumes.  The  monthly  volumes  reported  in  the  log  ranged  from  22-80  with 
an  average  of  55  ECGs  per  month.  After  interviews  with  staff,  it  was 
determined  that  the  reported  figures  account  for  60-70  percent  of  the 
volume  that  was  actually  done.  On  that  basis,  an  estimate  of  100  ECGs 
per  month,  or  1,200  per  year,  was  developed  for  use  in  the  evaluation. 

The  types  of  ECGs  were  determined  from  the  20  returned  data  collec¬ 
tion  forms.  Interviews  with  staff  were  used  to  confirm  whether  the  small 
sample  obtained  was  representative  of  their  typical  ECG  service.  The 
percentages  of  total  volume  by  type  of  ECG  used  for  the  evaluation  are 
summarized  in  Table  3-9. 
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These  volumes  and  types  of  ECGs  are  not  expected  to  change  significantly 
when  CAPOC  is  implemented. 


b.  Service  Effectiveness  Impact 

Data  from  the  X  Period  concerning  the  type  of  ECG  reader  and  the 
category  of  interpretation  are  summarized  in  Table  3-9;  these  data 
have  been  projected  for  full  implementation  of  CAPOC  at  this  site. 

These  data  are  combined  with  data  on  acceptable  and  preferred  turnaround 
times  for  different  types  of  ECGs  in  Table  3-10,  which  summarizes  the 
level  of  effective  ECG  services  at  Twentynine  Palms  during  the  X 
Period  (and  currently)  and  the  projected  impact  of  CAPOC  implementation. 
No  ECGs  met  effective  service  criteria  in  the  X  Period.  It  is  projected 
that  CAPOC  implementation  will  raise  the  level  of  effective  ECG  services 
to  95  percent.  This  projection  is  based  on  the  assumption  that  non¬ 
emergency  abnormal  ECGs  will  be  overread  by  physicians  at  Camp 
Pendleton  NRMC,  and  that  the  automatic  storage  and  retrieval  capability 
will  be  operational.  The  changes  in  service  effectiveness  will  result 
from  the  physicians  at  Twentynine  Palms  becoming  preferred  readers 
for  confirming  computer- generated  normal  ECG  interpretations,  and  the 
access  to  overreading  of  abnormal  ECGs  by  physicians  at  Camp  Pendleton 
NRMC. 


5.  Analysis  of  Qualitative  Data 

a.  Questionnaire  Administration 

In  the  X  Period,  staff  questionnaires  were  distributed  by  the  CAPOC 
representative  at  Camp  Pendleton  to  all  eight  physicians  at  Twentynine 
Palms  and  to  approximately  50  percent  of  other  staff  likely  to  use  ECGs. 
Questionnaires  were  returned  by  seven  physicians,  four  nurses,  and  27 
corpsmen. 

No  questionnaires  were  distributed  in  the  Y  Period  because  CAPOC 
had  not  been  implemented  at  that  time.  The  following  discussion  focuses 
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TABLE  3-9 


SUMMARY  OF  QUANTITATIVE  DATA  FOR 
TWENTYNINE  PALMS  MARINE  CORPS  BASE  BRANCH  HOSPITAL 


Volume 

lY£g. 

Routine  Outpatient 
Annual  Physical 
Routine  Inpatient 
Pre-Operative 
STAT/ER 

Reader 


Cardiologist 

Internist 

Other 


Interpretation 


Normal 

Abnormal 


X  Period 


1,200 


Projected 

1,200 


17.1% 

80.8% 
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on  the  results  of  questionnaires  distributed  in  the  X  Period  and  projec¬ 
tions  of  the  likely  changes  in  satisfaction  with  services  at  Twentynine 
Palms  when  CAPOC  is  operational. 

i.  ECG  Functions 

All  seven  staff  physicians  at  Twentynine  Palms  who  completed  ques¬ 
tionnaires  ordered  ECGs  to  be  taken  on  patients  and  interpreted  ECGs 
for  their  own  patients.  Three  also  interpreted  ECGs  for  patients  treated 
by  others.  Four  of  the  physician  respondents  had  held  their  current 
position  at  Twentynine  Palms  for  at  least  a  year,  three  for  less  than 
six  months.  It  should  be  noted  that  the  seven  physician  respondents 
represented  all  but  one  of  the  physician  staff  at  Twentynine  Palms. 


Service  Response  Times 


In  order  to  establish  what  levels  of  ECG  services  physicians  at 
Twentynine  Palms  considered  acceptable,  they  were  asked  about  response 
times  to  take  an  ECG  tracing  and  turnaround  times  to  receive  an  inter¬ 
pretation. 

Figures  3-24  and  3-25  show  the  distribution  of  responses  regarding 
the  acceptable  delays  between  an  order  for  a  STAT  ECG  and  between  the 
order  for  and  receipt  of  an  interpretation  for  a  STAT  ECG.  For  STAT 
ECGs  in  the  Emergency  Room,  all  seven  physician  respondents  agreed  that 
15  minutes  was  the  maximum  acceptable  delay  before  a  tracing  was  taken. 
Though  responses  regarding  the  delay  for  STAT  ECG  tracings  in  other  areas 
were  more  widely  distributed,  only  two  physicians  indicated  that  a  delay 
of  more  than  ?0  minutes  was  acceptable.  Acceptable  turnaround  times  for 
STAT  ECGs,  shown  in  Figure  3-25,  indicate  that  only  two  physicians  con¬ 
sidered  a  turnaround  time  of  more  than  15  minutes  acceptable. 

The  responses  regarding  service  response  times  for  routine  ECGs 
are  summarized  in  Figures  3-26  and  3-27.  Five  of  the  seven  respondents 
indicated  that  the  acceptable  delay  for  obtaining  the  tracing  was  between 
0  and  4  hours  and  the  acceptable  delay  for  return  of  the  interpretations 
was  1-24  hours. 
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No.  Respondents  3P  No.  Respondents 


E  3-24  ACCEPTABLE  DELAY  BETWEEN  REQUEST  FOR 
ST  AT  ECG  AND  TIME  IT  IS  TAKEN— PHYSICIAN 
RESPONDENTS  TO  STAFF  QUESTIONNAIRE  AT 
TWENTYNINE  PALMS  MARINE  CORPS  BASE 
BRANCH  HOSPITAL  -  X  PERIOD 


V/\  ST  AT  ECG 
0  ST  AT  ECG 


(minute*) 


FIGURE  3-2S  ACCEPTABLE  DELAY  BETWEEN  REQUEST  FOR 

STAT  ECG  AND  AVAILABILITY  OF  INTERPRETATION- 
PHYSICIAN  RESPONDENTS  TO  STAFF  QUESTIONNAIRE  AT 
TWENTYNINE  PALMS  MARINE  CORPS  BASE  BRANCH 
HOSPITAL  -  X  PERIOD 


1?'. 


in  E.R. 
in  other  areas 
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No.  Respondents 


<1 


1-4  5-8  9-24  >24 

(Hours) 

FIGURE  3-26  ACCEPTABLE  DELAY  BETWEEN  REQUEST  FOR 

ROUTINE  ECG  AND  TIME  IT  IS  TAKEN  -  PHYSICIAN 
RESPONDENTS  TO  STAFF  QUESTIONNAIRE  AT 
TWENTYNINE  PALMS  MARINE  CORPS  BASE  BRANCH 
HOSPITAL  -  X  PERIOD 


(Hours) 


FIGURE  3-27  ACCEPTABLE  DELAY  BETWEEN  REQUEST  FOR 

ROUTINE  ECG  AND  AVAILABILITY  OF  INTERPRETATION  - 
PHYSICIAN  RESPONDENTS  TO  STAFF  QUESTIONNAIRE  AT 
TWENTYNINE  PALMS  MARINE  CORPS  BASE  BRANCH 
HOSPITAL  -  X  PERIOD 


1.25 
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>1 


Figure  3-28  summarizes  the  results  derived  from  the  portion  of  the 
staff  questionnaire  concerning  the  importance  of  certain  attributes  of 
ECG  services  and  the  degree  of  satisfaction  with  the  performance  of  the 
current  ECG  system.  In  order  to  provide  a  basis  for  comparison,  responses 
have  been  weighted  as  follows: 


Importance  Ratin 


Very  important 


Satisfaction  Ratin 


Very  satisfied  1 


Ratine  Value 


Very  unimportant  5  Very  dissatisfied  5  -2 

The  sums  of  all  weighted  responses  to  each  question  are  displayed 
in  Figure  3-28.  The  maximum  possible  cumulative  rating  for  each  attri¬ 
bute  is  14  (7  physician  responses  times  +2  rating  value).  The  cumulative 
ratings  concerning  the  importance  of  ECG  service  attributes  are  shown  on 
the  left-hand  side  of  Figure  3-28;  those  concerning  satisfaction  with 
the  current  ECG  system  appear  on  the  right-hand  side. 

All  attributes  of  STAT  ECG  services  were  viewed  as  very  important, 
as  shown  in  Figure  3-28.  The  responses  indicate  that  physicians  were 
quite  satisfied  with  the  availability  of  ECG  tracings  from  the  current 
service,  but  only  marginally  satisfied  with  the  availability  of  inter¬ 
pretations  for  STAT  ECGs.  The  corresponding  features  of  routine  ECG 
services  were  rated  as  relatively  unimportant,  in  particular  24-hour 
coverage  for  tracings  and  interpretations.  Physicians  were  also  fairly 
dissatisfied  with  the  turnaround  time  for  interpretations  for  routine 
ECGs,  but  quite  satisfied  with  the  ability  of  the  current  service  to 
provide  tracings. 

Service  attributes  regarding  accuracy,  completeness,  and  ease  of 
understanding  interpretations  were  rated  by  all  seven  physician  respondents 
as  very  important,  as  were  the  technical  quality  of  the  ECG  tracing  and 
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the  availability  of  prior  tracings  for  serial  comparison.  All  of  these 
aspects  of  the  current  service  were  reported  to  be  only  marginally  sat¬ 
isfactory  as  shown  in  Figure  3-28.  Physicians  were  quite  dissatisfied 
with  the  availability  of  experts  to  discuss  the  significance  of  ECG  in¬ 
terpretations  in  patient  management. 

iv.  Recalls/Referrals 


Patients  sometimes  have  to  be  recalled  for  a  repeat  ECG  tracing 
because  the  quality  of  the  first  tracing  was  poor,  according  to  four 
physician  respondents  at  Twentynine  Palms.  Repeats  necessitated  by  loss 
of  the  ECG  were  not  reported,  but  four  physicians  indicated  that  they 
did  not  know  about  this  aspect  of  service.  All  seven  indicated  that 
patients  are  referred  to  other  medical  facilities,  and  three  reported 
that  these  referrals  are  often  necessary.  The  reasons  cited  were  lack 
of  a  cardiologist  and  the  need  for  specialized  cardiology  services  or 
a  second  opinion. 

v.  Other  Comments 

Other  physician  comments  regarding  X  Period  ECG  services  at 
Twentynine  Palms  were  solicited.  Recurring  comments  concerned  the  lack 
of  a  staff  cardiologist  and  the  age  of  the  ECG  equipment  at  the  site. 

vi.  Projected  CAP PC  Impacts 

CAPOC  should  result  in  an  improvement  in  the  level  of  satisfaction 
with  services  in  regards  to  relative  importance  of  the  attributes  as 
displayed  in  the  left  column  of  Figure  3-28.  Attributes  where  a  large 
discrepancy  was  noted  include: 

-  accuracy, 

-  completeness  of  interpretation, 

-  technical  quality  of  tracing, 

-  ease  of  reading  and  interpretation. 
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Satisfaction  with  Currant  Sarvica 


Importance 

ST  AT  ECO* 


Ability  to  taka  them  24  hri.  a  day 


Delay  between  request  and  taking 
of  tracing  is  acceptable 


24-hour  availability  of  interpretations 

Turnaround  time  for  interpretation 
is  acceptable 

Routine  ECGs 

Ability  to  take  them  24  hours  a  day 


Delay  between  request  and  taking 
of  tracing  is  acceptable 


24-hour  availability  of  interpretations 


Turnaround  time  for  interpretation 
is  acceptable 


ST  AT  ECGs 


Number  of  he  jrs  a  day  they  can  be  taken 


Delay  between  request  and  taking  of 
tracing 


Availability  of  interpretations 


Turnaround  time  for  interpretation 


Routine  ECGs 


Number  of  hours  a  day  they  can  be  taken 


Delay  between  request  and  taking  of 
tracing 


Availability  of  interpretations 


Turnaround  time  for  interpretation 


* 

$ 

i 


FIGURE  3-28  CUMULATIVE  RATING  OF  ECG  SERVICE  ATTRIBUTES  -  PHYSICIAN 

RESPONDENTS  TO  STAFF  QUESTIONNAIRE  AT  TWENTYNINE  PALMS  MARINE 
CORPS  BASE  BRANCH  HOSPITAL  -  X  PERIOD 
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Importance 


Satisfaction  with  Currant  Service 


Other  Service  Attribute* 


Other  Service  Attribute* 


Ability  to  take  ECG*  in  all  diagno«tic 
and  treatment  area* 


Ability  to  interpret  ECG*  in  all  diagnostic 
and  treatment  area* 


p3T=2 


+14 


taa+i 


ECG  interpretations  are  accurate 


+14 


ECG  interpretations  are  complete 
(all  aspects  noted) 


ECG  tracings  are  ot  high  technical 
quality 


P*-1 


+14  m 

Ef»V,y; 


+10 


Different  readers  make  same  readings 


+13  Eg. 


Interpretation*  are  easy  to  read  and 
understand 


Prior  tracings  are  available  for  com¬ 
parison  purposes 


Experts  ere  available  to  discuss  signi¬ 
ficance  of  ECG  interpretations  in 
patient  management 


+13  pa. 


+5 


Number  of  diagnostic  and  treatment  areas 
where  ECGs  can  be  taken 


Number  of  diagnostic  and  treatment  areas 
where  ECGs  can  be  interpreted 


Accuracy  of  ECG  interpretations 


Completeness  of  ECG  interpretations 
(all  aspects  noted) 


Technical  quality  of  ECG  tracings 


Consistency  of  interpretations  by  different 
readers 


Ease  of  reading  and  understanding  interpretations 


Availability  of  prior  tracings  for  comparison 
purposes 


Availability  of  experts  to  discuss  significance 
of  ECG  interpretations  in  patient  management 


FIGURE  3—28  (continued) 
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-  availability  of  experts  to  discuss  significance  of 
interpretation, 

-  availability,  and 

-  turnaround  time. 

CAPOC  will  enhance  completeness,  technical  quality,  and  availability. 
The  availability  of  unconfirmed  computer-generated  interpretations  should 
improve  accuracy.  The  availability  of  experts  to  discuss  significance 
of  ECG  interpretations  in  patient  management  should  increase  because 
physicians  at  Camp  Pendleton  NRMC  or  San  Diego  NRMC  can  obtain  a  copy 
of  the  ECG  tracing  immediately  when  called  by  a  physician  at  Twentynine 
Palms.  Previously,  the  physician  would  have  sent  the  ECG  or,  possibly, 
the  patient  to  one  of  these  facilities  for  an  evaluation. 

6.  Conclusions 


CAPOC  was  not  implemented  at  Twentynine  Palms  during  the  Y  Period 
data  collection.  When  CAPOC  is  implemented,  the  level  of  effective 
ECG  services  is  expected  to  be  improved  significantly.  The  improvements 
will  come  primarily  in  the  availability  of  effective  readers  at  the  site 
for  normal  ECGs  and  access  to  readers  at  Camp  Pendleton  NRMC  for  abnormal 
ECGs  • 
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E.  MIRAMAR  NAVAL  AIR  STATION  BRANCH  CLINIC 

1.  Introduction 


Miramar  Branch  Clinic  is  a  small  outpatient  clinic,  and  the  only  primary 
evaluation  site  that  lacks  inpatient  services.  Administratively,  Miramar 
is  a  component  of  the  San  Diego  Naval  Regional  Medical  Center,  but  it  oper¬ 
ates  ECG  services  quite  autonomously.  It  is  located  approximately  20  miles 
north  of  San  Diego. 

In  1979  (the  X  Period)  outpatient  visits  to  the  clinic  totalled  116,000. 
Of  these  49  percent  were  by  active-duty  military,  42  percent  were  by  depend¬ 
ents  of  active-duty  personnel,  and  the  remaining  9  percent  were  by  retirees 
and  their  dependents . 

In  1980  (the  Y  Period)  there  were  111,000  outpatient  visits.  The 
breakdown  was  59  percent  active-duty  military,  34  percent  dependents  of 
active-duty  personnel,  and  7  percent  retirees  and  their  dependents.  The 
primary  patient  population  came  from  the  Naval  Air  Station.  However,  due 
to  the  geographic  proximity  of  the  City  of  San  Diego,  patients  from  this 
area  also  used  the  clinic. 

Miramar  provides  routine  outpatient  care,  emergency  services,  and 
annual  flight  physicals.  Patients  requiring  more  specialized  outpatient 
care  or  inpatient  care  are  referred  to  San  Diego  NRMC. 

There  were  15  physicians  on  the  clinic  staff  in  both  periods,  six  of 
these  were  general  practitioners. 

2.  ECG  Service  Organization  and  Staffing 

The  process  for  obtaining  ECGs  did  not  change  with  the  implementation 
of  CAPOC.  In  both  the  X  and  Y  evaluation  periods,  the  majority  of  ECGs  were 
taken  in  two  locations — the  Flight  Examination  Clinic  and  the  Emergency  Room. 
Some  patients  who  were  having  flight  examinations  received  an  ECG  as  part 
of  these  examinations  (depending  upon  the  patient's  age  and  date  of  previous 
ECG).  If  an  ECG  was  indicated,  a  corpsman  took  the  tracing  and  included  it 
with  the  patient's  record  and  other  laboratory  test  results.  These  were 
reviewed  and  the  ECG  was  interpreted  when  the  flight  surgeon  examined  the 
patient;  the  interpretation  was  written  in  the  patient's  record. 
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STAT  ECGs  were  taken  in  the  Emergency  Room  by  corpsmen  assigned  to 
this  unit.  These  were  taken  either  in  response  to  a  physician's  order, 
or  to  obvious  or  clinical  symptoms,  such  as  chest  pain  or  irregular  heart 
rate.  STAT  ECGs  were  interpreted  by  the  attending  physician  and  the  in¬ 
terpretation  was  written  in  the  patient's  record. 

Routine  outpatient  ECGs  were  taken  in  either  location  or  the  ECG 
cart  was  moved  to  another  part  of  the  Branch  Clinic  if  indicated.  These 
tracings  were  interpreted  by  the  attending  physician  and  the  interpreta¬ 
tion  was  written  in  the  patient's  record. 

On  rare  occasions,  a  STAT  ECG  would  be  transported  by  automobile 
to  San  Diego  NRMC  for  interpretation  or  the  patient  and  ECG  would  be  re¬ 
ferred  to  San  Diego  by  ambulance  for  a  cardiology  consult.  For  abnormal 
ECGs  where  there  was  no  emergency  and  where  a  follow-up  examination  was 
indicated,  patients  were  referred  to  the  routine  Cardiology  Clinic  held 
in  San  Diego. 

After  CAPOC  was  implemented,  the  corpsman  attached  the  unconfirmed 
interpretation  to  the  ECG  tracing.  Thus,  the  physician  had  the  unconfirmed 
interpretation  available  when  he  saw  the  patient.  Confirmed  interpreta¬ 
tions  were  returned  the  next  morning  following  the  reading  and  editing 
at  San  Diego  NRMC. 

In  the  X  Period,  Miramar  had  a  single-channel  ECG  cart,  purchased 
in  1972,  and  a  three-channel  transmitting  cart,  purchased  in  1979  in  an¬ 
ticipation  of  CAPOC.  The  single-channel  cart  was  used  as  a  back-up. 

A  printer  was  added  after  CAPOC  services  were  installed. 

3.  Quantitative  Data  Collection  Methodology 

Data  collection  at  Miramar  was  limited  due  to  the  low  volume  of 
ECGs  taken  and  the  uncomplicated  ECG  services  provided  at  this  small 
Branch  Clinic.  In  total,  four  separate  site  visits  were  made  in  the  X 
Period  by  one  or  two  team  members.  During  these  visits,  information  was 
collected  on  ECG  types  and  volumes,  ECG  procedures,  turnaround  time,  use 
of  serial  comparison,  and  reading  staff  procedures.  The  information  was 
gained  through  a  review  of  log  books  and  reports  and  interviews  with  staff. 

One  site  visit  was  made  in  the  Y  Period  to  ascertain  the  status  of 
implementation,  distribute  staff  questionnaires,  interview  personnel,  and 
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collect  quantitative  data.  In  the  Y  Period,  much  of  the  quantitative 
data  was  available  from  the  computer-generated  workload  reports. 

4.  Quantitative  Data  Analysis 

a.  EC G  Volumes 


The  annual  ECG  volume  for  both  periods  was  1,320  (see  Table  3-11).  The 
estimate  for  the  Y  Period  was  based  on  an  analysis  and  projection  of  three 
months  of  volume  as  reported  by  the  CAPOC  system.  Transmission  problems 
had  prevented  ECGs  being  processed  by  CAPOC  until  August,  1980.  At  the 
time  of  Y  Period  data  collection,  the  problem  with  the  cart  still  existed 
and  some  ECGs  were  still  not  being  processed  by  CAPOC.  This  problem  must 
be  solved  if  full  utilization  is  to  be  achieved. 

b.  Service  Effectiveness  Impact 

Table  3-11  presents  results  for  ECG  reader  and  the  numbers  of  normal 
and  abnormal  interpretations. 

Table  3-12  presents  the  levels  of  ECG  service  effectiveness  for  the 
X  and  Y  Period  and  for  full  implementation.  No  ECGs  met  effective  ser¬ 
vice  criteria  during  the  X  Period.  In  the  Y  Period,  70  percent  of  all  ECGs 
met  effective  service  criteria.  The  change  was  due  to  the  general  prac¬ 
titioner  becoming  a  preferred  reader  of  normal  ECGs  with  the  back-up  of 
the  computer-generated  interpretation  and  access  to  specialists  at  San  Diego 
NKMC  for  overreading  of  abnormal  ECGs.  When  the  automatic  storage  and  re¬ 
trieval  of  prior  ECGs  becomes  operational  and  the  intermittent  problems  with 
the  cart  are  solved,  the  level  of  service  effectiveness  will  increase  to 
93  percent. 

5.  Analysis  of  Qualitative  Data 

No  physician  questionnaires  were  available  for  the  X  Period.  In  the 
Y  Period,  seven  physicians  at  Miramar  returned  completed  staff  question¬ 
naires;  four  of  the  respondents  performed  ECG  overreading  functions  at 
the  site  (these  are  referred  to  as  "readers").  Most  had  been  assigned 
to  Miramar  since  May,  1980.  Furthermore,  at  the  time  of  Y  Period  data 
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TABLE  3-11 


SUMMARY  OF  QUANTITATIVE  DATA  FOR 
MIRAMAR  NAVAL  AIR  STATION  BRANCH  CLINIC 


X  Period 

Y  Period 

.  » 

Volume 

1,320 

1,320 

Type 

Routine  Outpatient 

47% 

47% 

\' 

.  V 

Annual  Physical 

50% 

50% 

STAT/ER 

3% 

3% 

* 

Reader 

.  \ 

Cardiologist 

0 

29.1% 

Internist 

0 

0 

Other 

100% 

70.9% 

Interpretation 

Normal 

99% 

70% 

n 

Abnormal 

1% 

30% 

-- 
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SUMMARY  OF  IMPACTS  OF  CAPOC  ON  EFFECTIVENESS  OF  ECG  SERVICES 
AT  MIRAMAR  NAVAL  AIR  STATION  BRANCH  CLINIC 
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collection,  CAPOC  had  been  fully  operational  for  only  2  months.  There¬ 
fore,  physicians  had  not  yet  had  very  much  experience  with  CAPOC.  The 
following  analysis  is  focused  on  CAPOC  impacts  on  aspects  of  service  in 
which  physicians  had  gained  some  experience  and  on  the  perceptions  of 
the  possible  benefits  of  full  implementation  over  an  extended  period. 

The  technical  problems  experienced  with  transmission,  which  prevented 
CAPOC  use  for  many  months ,  were  mentioned  on  almost  all  responses  to 
the  questionnaire. 

Five  of  the  respondents  commented  on  the  impact  of  CAPOC  on  the 
amount  of  time  they  spent  each  week  reading/interpreting  ECGs.  Three 
of  these  (two  of  them  readers)  reported  that  CAPOC  had  saved  time,  citing 
the  assistance  of  CAPOC  in  interpreting  abnormal  ECGs  (one  response)  and 
general  time  saved  in  evaluating  each  ECG  (one  response).  Two  respondents 
reported  no  time  savings  (one  of  them  was  a  reader) ;  one  of  these  physi¬ 
cians  commented  that  each  ECG  still  had  to  be  read  manually  in  order  to 
verify  the  CAPOC  interpretation. 

Two  readers  commented  on  the  accuracy  of  CAPOC  vs.  manual  reading. 
They  agreed  that  CAPOC  was  more  accurate  for  rhythm  and  conduction  ab¬ 
normalities,  but  otherwise  their  responses  were  evenly  divided.  Four 
of  the  respondents  reported  that  computer  errors  tended  to  be  false/ 
positive  results;  one  believed  that  false/negative  results  were  more 
likely . 

Physicians  rated  their  overall  satisfaction  with  CAPOC  as  follows: 


Extremely  satisfied  0 
Very  satisfied  1 
Fairly  satisfied  4 
Somewhat  unsatisfied  1 
Very  unsatisfied  0 
Do  not  know  1 


Free-form  comments  indicated  that  most  physicians  believed  CAPOC 
would  be  beneficial  when  the  system  reliability  was  improved.  The 
availability  of  ECG  interpretations  24  hours  per  day  was  cited  as  a 
major  benefit  of  the  system.  In  addition  to  the  technical  difficulties 
alluded  to  previously,  two  physicians  believed  that  technicians  required 
further  training  in  the  use  of  the  CAPOC  system. 
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6.  Conclusions 


CAPOC  has  greatly  improved  the  ECG  services  at  Miramar  Branch  Clinic. 
Technical  problems  still  existed  at  the  time  of  Y  Period  data  collection 
and  were  impeding  full  utilization  of  the  system  and  staff  acceptance. 

When  the  system  is  fully  implemented  at  this  site,  93  percent  of  the  ECGs 
will  meet  criteria  for  effective  services. 
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CHAPTER  4.  SERVICE  EFFECTIVENESS— SECONDARY  SITES 


£ 
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A. _ USAF  HOSPITAL,  VANDENBERG  AIR  FORCE  BASE 

The  USAF  Hospital  at  Vandenberg  Air  Force  Base  is  a  40-bed  hospital 
operating  at  75  percent  occupancy.  It  is  located  near  the  town  of 
Lompoc,  California,  approximately  30  miles  north  of  Santa  Barbara. 
Vandenberg  provides  general  medical  and  surgical  care  and  has  a  seven- 
bed  ICU.  Patients  with  complex  diagnostic  or  treatment  problems  are 
usually  referred  to  San  Diego  NRMC,  Letterman  Army  Base,  or  Santa  Monica 
(a  civilian  hospital  in  Santa  Barbara).  Outpatient  visits  numbered 
approximately  150,000  annually  in  1979.  In  the  X  Period,  the  staff 
included  one  civilian  cardiologist,  two  internists,  and  15  other 
physicians . 

In  the  X  Period,  the  ECG  Laboratory  was  part  of  the  Family  Practice 
Department  and  was  staffed  by  three  technicians  who  shared  one  equiva¬ 
lent  full-time  position.  Each  technician  spent  one  week  at  a  time 
taking  ECGs  and  administering  treadmill  examinations.  During  the 
other  2  weeks,  the  technicians  held  other  assignments  in  the  Family 
Practice  Department.  Approximately  80  percent  of  their  time  in  the 
ECG  Laboratory  was  spent  directly  on  ECG  responsibilities. 

ECGs  were  taken  by  the  ECG  Laboratory  staff  between  0730  and  1630 
on  Monday  through  Friday.  On  nights  and  weekends,  ECGs  were  taken  in 
the  ER,  ICU,  or  on  the  inpatient  wards  by  a  corpsman  assigned  to  those 
units.  There  were  two  single-channel  carts:  one  in  the  ECG  Laboratory 
and  one  on  an  inpatient  ward.  CAP0C  had  not  been  implemented  at 
Vandenberg  during  the  Y  Period  data  collection  due  to  transmission  and 
ECG  cart  problems.  CAPOC  equipment  will  include  a  three-channel  trans¬ 
mitting  ECG  cart  and  printer. 

Table  4-1  summarizes  the  quantitative  results  for  this  site.  The 
annual  ECG  volume  during  the  X  Period  was  2,808.  This  volume  is  not  ex¬ 
pected  to  be  impacted  by  CAPOC  implementation.  Table  4-2  summarizes  the 
level  of  ECG  service  effectiveness  in  the  X  Period  (and  currently)  and  the 
projected  impact  of  CAPOC  implementation  on  these  services.  Effective  ECGs 
accounted  for  57  percent  of  all  ECGs  in  the  X  Period.  When  CAPOC  is  ' 
implemented,  this  level  will  increase  to  76.5  percent.  The  increase 
will  occur  because  internists  become  preferred  readers  for  confirming 


TABLE  4-1 


SUMMARY  OF  QUANTITATIVE  DATA  FOR 
USAF  HOSPITAL,  VANDENBERG  AIR  FORCE  BASE 


Volume 


X  Period  Projected 

2,808  2,808 


Ixpe 

Routine  Outpatient 
Annual  Physical 
Routine  Inpatient 
Pre-Operative 
STAT/ER 


44% 

44% 

19% 

19% 

14% 

14% 

12% 

12% 

11% 

11% 

Reader 

Cardiologist 

Internist 

Other 


47% 

27% 

26% 


46.9  % 
27.0% 
26.1% 


Interpretation 

Normal 

Abnormal 


59%  59% 

41%  41% 
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*Not  implemented  during  Y  Period  data  collection. 


normal  ECG  interpretations  produced  by  the  computer.  The  proportion  of 
ECG  services  meeting  criteria  for  preferred  services  will  increase  from 
8  percent  in  the  X  Period  to  30  percent  for  full  implementation. 

Conclusions :  CAPOC  was  not  operational  at  Vandenberg  during  the  Y 
Period  data  collection.  Effective  ECG  services  are  expected  to  in¬ 
crease  when  CAPOC  is  implemented. 
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B.  USAF  HOSPITAL,  EDWARDS  AIR  FORCE  BASE 


Edwards  Air  Force  Base  has  a  25-bed  hospital  operating  at  80  per¬ 
cent  occupancy.  It  is  located  near  the  town  of  Rosamond,  California, 
and  35  miles  from  the  closest  referral  hospital  in  Lancaster,  California. 
Depending  upon  the  need  for  and  availability  of  services,  patients  from 
Edwards  may  be  referred  to  major  civilian  and  military  hospitals  in 
southern  California. 

This  small  hospital  has  an  Acute  Care  Unit,  which  is  used  for 
close  observation  or  for  stabilizing  critical  patients  prior  to  being 
transported  to  a  referral  hospital.  There  were  14  physicians  on  the 
hospital  staff  at  Edwards  Air  Force  Base  in  both  the  X  and  Y  Periods. 

One  of  them  was  an  internist. 

The  ECG  process  at  Edwards  has  not  changed  as  a  result  of  CAPOC 
implementation.  Approximately  20  technicians  took  ECGs  in  six  locations. 
Edwards  has  six  ECG  machines .  One  three-channel  cart  located  in  the 
Treatment  Room  that  was  purchased  in  1978  in  anticipation  of  CAPOC  was 
used  for  taking  most  ECGs;  the  other  five  carts  were  single-channel 
models  located  in  an  inpatient  ward,  the  Emergency  Room,  the  area  in  which 
annual  physical  examinations  are  given,  and  two  clinics.  There  was  a 
CAPOC  printer  in  the  Treatment  Room.  In  the  Treatment  Room,  most  of 
the  ECGs  were  taken  between  0730  and  1630,  Monday  through  Friday.  On 
nights  and  weekends,  ECGs  were  either  taken  in  the  Emergency  Room  or  on 
an  inpatient  ward. 

Table  4-3  presents  data  on  ECG  volumes,  readers,  and  type  of  ECG  inter¬ 
pretation.  The  annual  ECG  volume  was  3,780  ECGs  in  both  the  X  and  Y  Periods. 
In  the  Y  Period,  only  37  percent  of  all  ECGs  were  being  done  by  CAPOC.  ECGs 
done  as  part  of  annual  physicals  were  not  being  captured  because  there  was 
no  three-channel,  transmitting  cart  in  the  flight  sureeon's  office. 

Table  4-4  presents  a  summary  of  ECG  service  effectiveness  for  the 
X  and  Y  Periods  and  for  full  implementation.  One  quarter  of  all  base¬ 
line  ECGs  met  criteria  for  effective  ECG  services.  This  increased  to 
28  percent  in  the  Y  Period.  The  slight  increase  was  due  to  the  general 
practitioners  who  read  ECGs  done  in  the  Emergency  Room  becoming  preferred 
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TABLE  4-3 


SUMMARY  OF  QUANTITATIVE  DATA  FOR 
USAF  HOSPITAL,  EDWARDS  AIR  FORCE  BASE 


Volume 


X  Period  Y  Period 

3,780  3,780 


Type 

Routine  Outpatient 
Annual  Physical 
Routine  Inpatient 
Pre- Operative 
STAT/ER 


10% 

10% 

63% 

63% 

10% 

10% 

7% 

7% 

10% 

10% 

Reader 

Cardiologist 

Internist 

Other 


0 

32% 

68% 


0 

32% 

68% 


Interpretation 

Normal 

Abnormal 


77%  77% 

23%  23% 
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readers  when  confirming  normal  ECG  interpretations  generated  by  the  com¬ 
puter.  If  the  63  percent  of  ECGs  done  in  the  flight  surgeon's  office 
were  processed  by  CAPOC,  the  level  of  effective  service  would  increase 
to  78  percent.  This  change  would  require  that  one  additional  three- 
channel  ECG  cart  and  printer  be  purchased  for  the  flight  surgeon's 
office. 

Physicians  at  Edwards  like  the  rapid  availability  of  an  unconfirmed 
interpretation,  especially  on  nights  and  weekends.  Due  to  the  high  turn¬ 
over  rate  of  corpsmen  and  the  large  number  who  take  ECGs,  insufficient 
training  on  CAPOC  use  was  cited  as  a  problem. 

Conclusions :  At  Edwards  Air  Force  Base  Hospital,  effective  ECG  services 
increased  only  slightly  in  the  Y  Period  because  the  majority  of  ECGs 
were  still  being  done  in  a  manual  mode.  The  automated  system  will  have 
a  significant  impact  on  services  when  ECGs  done  as  part  of  annual  physi¬ 
cals  are  processed  by  CAPOC. 
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George  Air  Force  Base,  located  near  Victorville,  California,  approxi¬ 
mately  50  miles  north  of  San  Bernardino,  has  a  small  42-bed  hospital. 

The  closest  and  most  frequently  used  referral  hospital  is  at  March  Air 
Force  Base.  Patients  are  occasionally  transported  to  other  large  hospi¬ 
tals  in  southern  California. 

The  hospital  has  a  two-bed  Close  Observation  Room  for  observing 
and  monitoring  critically  ill  patients.  In  both  the  X  and  Y  Periods, 
the  hospital  staff  included  15  physicians,  two  of  whom  were  internists. 

The  ECG  process  did  not  change  as  a  result  of  CAPOC  implementation. 
There  were  two  corpsmen/ technicians  who  took  most  of  the  ECGs  in  the 
Treatment  Room,  which  was  part  of  the  Primary  Care  service,  one  technician 
in  the  Flight  Examination  Department,  and  approximately  10  other  corpsmen 
who  took  some  ECGs  on  the  wards  and  in  the  Emergency  Room.  ECGs  were 
taken  from  0730  to  1630  on  Monday  through  Friday  before  and  after  auto¬ 
mation.  Most  ECGs  were  taken  in  the  morning.  ECG  technicians  were 
assigned  other  duties  when  not  taking  ECGs. 

In  the  X  Period,  there  were  four  single-channel  ECG  carts  at  George. 
George  acquired  a  three-channel  transmitting  ECG  cart  and  printer  for 
CAPOC.  The  printer  was  located  in  the  Treatment  Room. 

Basic  data  on  volume  and  type  of  ECGs  and  ECG  readers  are  presented 
in  Table  4-5.  The  annual  ECG  volume  in  the  X  Period  was  1,332  ECGs.  This 
volume  had  increased  by  14  percent  to  1,496  ECGs  in  the  Y  Period.  Not 
all  ECGs  were  being  processed  by  CAPOC  at  the  time  of  the  Y  Period  site 
visit.  Part  of  the  ECGs  done  in  the  inpatient  wards  and  during  annual 
physicals  were  still  being  processed  with  a  manual  system.  There  were  no 
telephone  jacks  in  some  areas  of  the  inpatient  wards,  making  CAPOC  access 
impossible.  ECGs  done  as  part  of  annual  physicals  were  not  done  by  CAPOC 
for  administrative  reasons;  this  is  expected  to  be  changed  soon  so  that 
all  these  ECGs  can  be  done  by  CAPOC. 

Table  4-6  summarizes  the  levels  of  ECG  service  effectiveness  in 
the  X  and  Y  Periods  and  for  full  implementation.  The  overall  level  of 
ECG  service  effectiveness  did  not  change  with  the  implementation  of 
CAPOC,  nor  is  it  projected  to  change  when  all  ECGs  are  done  by  CAPOC 
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TABLE  4-5 

SUMMARY  OF  QUANTITATIVE  DATA  FOR 
USAF  HOSPITAL,  GEORGE  AIR  FORCE  BASE 


Volume 

Tyge 

Routine  Outpatient 
Annual  Physical 
STAT/ER 
Pediatric 

Reader 

Cardiologist 

Internist 

Other 


X  Period 


1,332 


Y  Period 


1,496 


Interpretation 


Normal 

Abnormal 
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7 


■'■1 


839  (63.0)  940  (62.9)  941  (62.9) 


upon  full  implementation.  However,  the  level  of  preferred  service  will 
change  due  to  CAPOC  implementation.  In  the  X  Period,  no  ECGs  met  prefer¬ 
red  service  criteria.  In  the  Y  Period,  12  percent  met  preferred  criteria, 
and  it  is  projected  that  21  percent  will  meet  preferred  criteria  upon  full 
implementation.  This  change  is  due  to  all  readers  becoming  preferred 
readers  for  confirming  the  computer-generated  interpretation  of  normal 
ECGs . 

The  physicians  at  George  like  the  CAPOC  system,  especially  the 
rapid  availability  of  an  unconfirmed  interpretation.  They  also  believe 
the  quality  of  tracings  has  improved,  partly  due  to  CAPOC  and  partly  due 
to  the  use  of  three-channel  ECG  machines. 

Conclusions :  CAPOC  has  not,  and  is  not,  expected  to  increase  overall 
ECG  service  effectiveness  at  George.  CAPOC  has  impacted  the  level  of 
preferred  services,  and  is  perceived  as  having  additional,  unquantified 
benefits. 
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D.  USAF  MEDICAL  FACILITY,  NORTON  AIR  FORCE  BAS E 
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The  USAF  Medical  Facility  at  Norton  Air  Force  Base  is  a  small 
outpatient  clinic  located  in  urban  San  Bernardino,  California.  For 
inpatient  care,  patients  are  referred  to  either  March  Air  Force  Base 
or  to  local  civilian  hospitals.  In  both  the  X  and  Y  Periods,  the  hos¬ 
pital  staff  included  13  physicians,  most  of  whom  were  either  General 
Practitioners  or  Flight  Surgeons. 

The  ECG  process  at  Norton  has  not  changed  due  to  CAPOC  implemen¬ 
tation.  ECGs  were  done  primarily  by  one  technician  who  also  had  other 
responsibilities.  Four  other  technicians  also  knew  how  to  take  ECGs. 
Nearly  all  of  the  ECGs  were  taken  in  the  ECG  Laboratory,  which  operated 
from  0800  to  1630,  Monday  through  Friday.  A  small  percentage  of  the 
ECGs  were  taken  in  the  Emergency  Room.  The  clinic,  including  the 
Emergency  Room,  was  closed  on  nights  and  weekends.  Emergency  services, 
including  ECGs,  were  obtained  at  the  referral  hospitals. 

In  the  X  Period,  there  was  one  three-channel  non-transmitting 
cart  in  the  ECG  Laboratory  and  a  single-channel  ECG  cart  in  the  ER. 

Norton  acquired  a  three-channel,  transmitting  ECG  cart  and  printer  for 
use  with  CAPOC.  Both  pieces  of  equipment  were  placed  in  the  ECG  Labora¬ 
tory.  The  non-transmitting  cart  was  kept  for  back-up. 

Basic  quantitative  results  are  summarized  in  Table  4-7.  The  annual 
ECG  volume  in  the  X  Period  was  2,880  (as  estimated  from  log  books  main¬ 
tained  in  the  ECG  Laboratory).  This  volume  decreased  to  2,426  in  the  Y 
Period.  Only  63  percent  of  all  ECGs  done  at  Norton  were  being  processed 
by  CAPOC  at  the  time  of  Y  Period  data  collection.  There  were  still 
intermittent  problems  with  the  telephone  lines,  such  as  being  unable  to 
get  through  and  being  disconnected  in  the  middle  of  a  transmission. 

Table  4-8  presents  a  summary  of  the  levels  of  effective  ECG  services 
in  the  X  and  Y  Periods  and  projections  of  effectiveness  when  CAPOC  is 
fully  implemented.  No  ECGs  met  effective  service  criteria  in  the  X  Period 
because  there  were  no  cardiologists  or  internists  at  Norton.  With  CAPOC, 
any  reader  is  considered  a  preferred  reader  for  confining  normal  ECG 
interpretations  generated  by  the  computer.  This  change  resulted  in  40 
percent  of  all  ECGs  meeting  effective  service  criteria  in  the  Y  Period. 
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TABLE  4-7 


SUMMARY  OF  QUANTITATIVE  DATA  FOR 
USAF  HOSPITAL,  NORTON  AIR  FORCE  BASE 


X  Period 


Y  Period 


Volume 


2,280 


2,426 


Type 

Routine  Outpatient 
Annual  Physical 
Routine  Inpatient 
STAT/ER 

Reader 


Cardiologist 

Internist 

Other 


77.6% 

22.4% 


Interpretation 


Normal 

Abnormal 
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This  level  will  increase  to  63  percent  when  all  ECGs  are  done  by  CAPOC. 
This  increase  at  full  implementation  will  be  a  result  of  capturing  the 
additional  37  percent  of  ECGs  taken  at  Norton. 

Physicians  at  Norton  report  that  they  do  not  like  the  CAPOC  system. 
They  report  a  very  low  level  of  agreement  between  the  unconfirmed  report 
and  their  own  interpretation.  They  feel  that  the  disagreement  can,  in 
part,  be  explained  by  the  technical  problems  in  interfacing  their  trans¬ 
mitting  cart,  communication  lines,  and  the  CAPOC  system. 

Conclusions :  Despite  only  63  percent  of  all  ECGs  being  done  by  CAPOC 
in  the  Y  Period,  CAPOC  has  had  a  major  impact  on  the  number  of  ECGs 
meeting  evaluation  criteria  at  Norton.  This  effect  will  be  even  greater 
when  all  ECGs  are  done  by  CAPOC.  Technical  difficulties  may  have  resulted 
in  dissatisfaction  with  the  system.  This  issue  needs  further  investiga¬ 
tion  and  resolution. 
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E.  USAF  REGIONAL  HOSPITAL,  MARCH  AIR  FORCE  BASE 


The  USAF  Regional  Hospital  at  March  Air  Force  Base  (March)  is  a 
medium-sized,  125-bed  hospital  located  in  Riverside,  California.  In 
addition  to  providing  a  broad  range  of  health  care  services  to  its  local 
patient  population,  this  hospital  serves  as  a  referral  hospital  for 
other  small  military  hospitals  and  clinics  in  the  region.  Patients  with 
problems  that  cannot  be  handled  at  March  are  referred  to  San  Diego  NRMC 
or  Loma  Linda  University  Medical  Center. 

For  delivery  of  acute  cardiology  care,  March  has  a  10-bed  combined 
cardiac  care  unit  and  intensive  care  unit.  Pediatric  cardiology  care  is 
provided  by  a  cardiologist  from  San  Diego  NRMC  who  visits  March  once 
each  month.  In  the  X  Period,  there  were  five  internists  who  shared  ECG 
reading  responsibilities.  This  number  decreased  to  four  in  the  Y  Period. 
One  internist  was  scheduled  to  read  routine  ECGs  and  one  was  assigned 
STAT  ECGs  each  day  in  both  periods. 

The  ECG  process  and  technician  staffing  did  not  change  between  the 
X  and  Y  Periods.  Four  technicians  who  did  most  of  the  ECGs  were  located 
in  the  Cardiopulmonary  (CP)  Laboratory.  They  each  had  other  responsi¬ 
bilities  in  addition  to  those  related  to  ECGs.  The  CP  Laboratory  staff 
took  ECGs  from  0730  to  1630,  Monday  through  Friday.  At  night  and  on 
weekends,  ECGs  were  taken  in  the  Emergency  Room,  CCU,  and  inpatient  wards. 

In  the  X  Period,  the  ECG  equipment  consisted  of  one  three-channel, 
non- transmit  ting  ECG  cart  located  in  the  CP  Laboratory  and  seven  single¬ 
channel  ECG  carts  located  in  the  CP  Laboratory,  Emergency  Room,  inpatient 
wards,  and  CCU.  In  the  Y  Period,  March  acquired  a  three-channel  trans¬ 
mitting  ECG  cart  and  printer  for  CAPOC  while  retaining  all  equipment  in 
place  previously.  The  CAPOC  equipment  was  placed  in  the  CP  Laboratory. 

The  three-channel  non-transmitting  cart  was  devoted  to  stress  testing. 

Table  4-9  presents  data  on  volumes  and  types  of  ECGs  and  on  ECG 
reader.  The  annual  ECG  volume  in  both  X  and  Y  Periods  was  estimated  to 
be  6,000  ECGs.  Only  routine  outpatient,  pre-operative  and  Emergency  Room 
ECGs  were  done  by  CAPOC  during  Y  Period  data  collection  because  only 
one  transmitting  cart  in  the  CP  Laboratory  was  available.  (This  cart  was 
also  available  for  use  in  the  Emergency  Room.) 
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TABLE  4-9 


SUMMARY  OF  QUANTITATIVE  DATA  FOR 
USAF  REGIONAL  HOSPITAL,  MARCH  AIR  FORCE  BASE 


X  Period 

Y  Period 

Volume 

6,000 

6,000 

Type 

Routine  Outpatient 

40% 

40% 

Annual  Physical 

21% 

21% 

Routine  Inpatient 

16% 

16% 

Pre-Operative 

14% 

14% 

STAT/ER 

8% 

8% 

Pediatric 

1% 

1% 

Reader 

Cardiologist 

0 

0 

Internist 

100% 

100% 

Other 

0 

0 

Interpretation 

Normal 

72.5% 

72% 

Abnormal 

27.5% 

28% 

156 


Arthur  D  Little  Inc 


TABLE  4-10 

SUMMARY  OF  IMPACTS  OF  CAPOC  ON  EFFECTIVENESS  OF  ECG  SERVICES 
AT  USAF  REGIONAL  HOSPITAL,  MARCH  AIR  FORCE  BASE 
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5,046  (84.1) 


Table  4-10  summarizes  the  effectiveness  of  ECG  services  in  the  X 
and  Y  Periods  and  for  full  implementation.  In  the  X  Period,  83  percent 
of  all  ECGs  met  effective  service  criteria.  This  changed  slightly  to 
84  percent  in  the  Y  Period.  The  lack  of  significant  change  is  attri¬ 
buted  to  the  fact  that  only  62  percent  of  all  ECGs  were  being  captured 
by  CAPOC.  Full  implementation  projections  are  based  on  the  purchase  of 
an  additional  three-channel  transmitting  ECG  cart  to  capture  ECGs  done 
in  the  inpatient  areas  and  having  ECGs  taken  as  part  of  annual  physicals 
done  by  CAPOC.  These  changes  would  not  increase  the  level  of  effective¬ 
ness  over  the  Y  Period  level.  However,  the  percent  of  ECGs  meeting 
criteria  for  the  preferred  level  of  service  would  increase  from  6  per¬ 
cent  to  31  percent.  This  increase  would  be  due  to  internists  becoming 
preferred  readers  for  normal  ECGs. 

Conclusions :  Not  all  ECGs  done  at  March  are  being  catpured  by  CAPOC. 
CAPOC  has  not  resulted,  nor  is  it  expected  to  result,  in  significant 
changes  in  the  level  of  effective  ECG  services  at  this  site. 
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F.  LONG  BEACH  NAVAL  REGIONAL  MEDICAL  CENTER 


Long  Beach  Naval  Regional  Medical  Center  is  a  regional  hospital 
with  129  authorized  beds,  which  provides  general  medical  and  surgical 
services.  The  typical  occupancy  rate  is  80  percent.  The  facility  has 
an  ICU  with  about  10  beds.  The  hospital  has  many  specialty  outpatient 
clinics  and  outpatient  visits  total  approximately  510,000  in  the  X  Period. 
The  staff  that  reads  ECGs  consists  of  one  cardiologist  and  five  internists. 

During  the  X  Period  the  ECG  Laboratory  was  located  in  the  Internal 
Medicine  Clinic  and  was  staffed  by  five  full-time  technicians.  There 
was  one  cardiologist  on  the  hospital  staff  during  the  X  Period.  CAPOC 
was  not  implemented  at  Long  Beach  NRMC  during  the  Y  Period  data  collec¬ 
tion  because  of  delays  in  installing  the  required  telephone  interface. 

In  the  X  Period,  routine  ECGs  were  taken  on  Monday  through  Friday 
between  0800  and  1615  hours.  On  nights  and  weekends,  ECGs  were  done 
in  the  Emergency  Room  and  Intensive  Care  Unit.  These  ECGs  were  usually 
done  by  corpsmen  and  nurses  working  in  these  units  who  had  been  trained 
in  the  taking  of  ECGs.  For  the  inpatient  wards,  an  ECG  technician  from 
the  Internal  Medicine  Clinic  was  on-call  24  hours  a  day  to  take  ECGs. 

ECG  equipment  consisted  of  six  carts  (four  three-channel  transmitting 
carts  and  two  single-channel  carts) ,  which  were  located  in  the  ECG  Labora¬ 
tory  and  in  patient  care  areas.  When  CAPOC  is  implemented.  Long  Beach 
NRMC  will  have  an  overread  station  located  in  the  ECG  Laboratory.  They 
will  be  able  to  perform  all  functions  related  to  editing,  retrieval,  etc. 

A  summary  of  quantitative  data  for  this  site  is  given  in  Table  4-11. 
The  annual  ECG  volume  in  the  X  Period  was  5,280  ECGs.  This  volume  is  not 
expected  to  change  dramatically  with  CAPOC  implementation.  Table  4-12 
summarizes  the  level  of  ECG  service  effectiveness  in  the  X  Period  (and 
currently,  without  CAPOC)  and  when  CAPOC  is  implemented.  In  the  X  Period, 
96  percent  of  all  ECGs  met  effective  service  criteria.  This  is  only 
expected  to  increase  to  98  percent  with  CAPOC  implementation  when 
the  automated  storage  and  retrieval  of  prior  ECGs  becomes  operational. 
While  the  overall  effectiveness  will  not  change  significantly,  the 
percentage  of  ECGs  meeting  preferred  criteria  will  increase  from  2  per¬ 
cent  in  the  X  Period  to  90  percent  with  the  CAPOC  system.  This  shift 

159 


Arthur  D  Little  Inc 


TABLE  4-11 

SUMMARY  OF  QUANTITATIVE  DATA  FOR 
LONG  BEACH  NAVAL  REGIONAL  MEDICAL  CENTER 
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★Not  implemented  during  Y  Period  data  collection. 


is  due  to  changes  in  the  definition  of  preferred  readers  for  normal  ECGs, 
improvements  in  turnaround  time,  and  use  of  serial  comparisons. 

Conclusions :  CAPOC  was  not  implemented  at  Long  Beach  NRMC  in  the  Y 
Period  data  collection.  CAPOC  is  expected  to  increase  the  level  of 
preferred  services  but  not  significantly  affect  the  baseline  effectiveness 
of  services  at  Long  Beach. 
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G.  NAVAL  STATION  SAN  DIEGO  BRANCH  CLINIC 


The  Naval  Station  San  Diego  Branch  Clinic  provides  only  general 
outpatient  medical  care.  Patients  requiring  hospitalization  or  special¬ 
ized  diagnostic  or  therapeutic  services  are  referred  to  San  Diego  NRMC. 

In  the  X  Period,  the  staff  consisted  of  nine  physicians — one  internist, 
one  family  practitioner,  one  occupational  medicine  specialist,  one 
psychiatrist,  and  five  civilian  general  practitioners.  One  general 
practitioner  and  the  family  practitioner  left  between  the  X  and  Y  Periods 
so  the  Y  Period  staffing  was  reduced  to  seven. 

The  ECG  Laboratory,  in  the  Physical  Examination  Clinic,  was  staffed 
by  one  full-time  technician  in  both  periods.  Routine  ECGs  were  taken 
between  0730  and  1600  on  Monday  through  Friday.  On  nights  and  weekends, 
patients  were  sent  to  San  Diego  NRMC  for  ECGs  when  required.  In  the  X 
Period,  there  was  a  single-channel  ECG  machine  in  the  ECG  Laboratory. 

In  the  Y  Period,  the  equipment  consisted  of  a  three-channel  transmitting 
cart  and  CAPOC  printer  in  the  ECG  Laboratory,  and  a  three-channel  ECG 
cart  in  the  Emergency  Room.  The  old  single-channel  ECG  cart  was  kept 
for  back-up.  In  the  X  Period,  the  ECG  tracings  were  sent  by  guard  mail 
to  San  Diego  NRMC  to  be  read.  The  tracing  and  interpretation  were  re¬ 
turned  from  two  to  seven  days  later. 

Basic  data  on  ECG  volumes  and  types  and  on  ECG  readers  are  summarized 
in  Table  4-13.  The  annual  ECG  volume  in  the  X  Period  was  3,148.  In  the  Y 
Period,  the  annual  ECG  volume  decreased  by  14  percent  to  2,712.  This  de¬ 
crease  might  be  due  to  the  decrease  in  physician  staff  that  occurred 
between  the  two  periods. 

Table  4-14  summarizes  the  effectiveness  of  ECG  services  at  the 
Naval  Station  in  the  X  and  Y  Periods  and  for  full  implementation.  In 
the  X  Period,  69  percent  of  all  ECGs  met  effective  service  criteria. 

This  number  increased  to  80  percent  in  the  Y  Period.  The  increase  is 
due  to  improved  turnaround  time  for  return  of  ECG  interpretations.  When 
automated  storage  and  retrieval  of  prior  ECGs  is  available,  the  effective¬ 
ness  of  service  is  expected  to  increase  to  96  percent  because  of  an  in¬ 
crease  in  the  use  of  serial  comparison  for  abnormal  tracings. 

The  staff  at  the  Naval  Station  likes  the  CAPOC  system.  The  most 
important  feature  for  them  is  the  reduced  turnaround  time,  especially 
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TABLE  4-13 


SUMMARY  OF  QUANTITATIVE  DATA  FOR 
NAVAL  STATION  SAN  DIEGO  BRANCH  CLINIC 


X  Period 

Y  Period 

Volume 

3,148 

2,712 

Ty.Eg. 

Routine  Outpatient 

20% 

20% 

Annual  Physical 

80% 

80% 

Reader 

Card iologist 

100% 

20% 

Internist 

0 

0 

Other 

0 

80% 

Interpretation 

Normal 

80% 

80% 

Abnormal 

20% 

20% 
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for  unconfirmed  interpretations,  which  they  feel  improves  patient  care. 
Having  the  corpsmen  screen  normal  and  abnormal  ECGs  also  saves  time  for 
the  physicians. 

Conclusions :  CAPOC  has  improved  ECG  service  effectiveness  at  the  Naval 
Station  because  of  improved  turnaround  time.  Availability  of  prior 
ECGs  for  serial  comparison  will  result  in  an  even  greater  improvement . 
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H.  NAVAL  AIR  STATION  NORTH  ISLAND  BRANCH  CLINIC 


The  Naval  Air  Station  North  Island  is  located  on  the  northern  tip 
of  Coronado,  across  the  San  Diego-Coronado  Bay  Bridge  from  San  Diego. 

The  Branch  Clinic  provides  outpatient  and  emergency  services  primarily 
for  the  active  duty  military  md  civilian  workforce  assigned  to  the  faci¬ 
lity.  There  are  four  specialty  clinics  -  Occupational  Health,  General 
Military,  Physical  Examination  and  Ophthalmology.  When  other  services 
are  required,  patients  are  referred  to  San  Diego  NRMC.  When  operating 
with  non-automated  services  the  clinic  was  staffed  by  12  physicians — 
one  occupational  medicine  specialist,  10  flight  surgeons  and  one  general 
medical  officer.  In  the  X  Period  there  were  64,000  outpatient  visits 
to  the  clinics. 

Before  automation,  there  was  no  formal  ECG  Laboratory  at  North  Is¬ 
land.  A  three-channel  Marquette  Cart  with  a  cassette  recorder  resided 
in  the  Physical  Examination  Clinic  but  was  moved  to  wherever  it  was  re¬ 
quired.  The  cart  was  on  loan  from  San  Diego  NRMC.  A  single-channel 
HP  cart  was  used  as  a  backup  and  to  run  ECG  rhythm  strips.  All  corps- 
men  were  trained  to  take  ECGs.  ECGs  were  taken  from  0730  to  1600  on 
Monday  through  Friday.  The  corpsman  on  duty  in  the  emergency  room  was 
responsible  for  ECGs  required  outside  these  hours.  In  the  X  Period,  the 
annual  ECG  volume  at  North  Island  was  3149  (Table  4-15).  The  CAPOC  system 
was  not  operational  at  North  Island  during  the  Y  Period  data  collection 
effort  conducted  in  the  fall  of  1980.  This  was  attributed  primarily  to 
malfunctions  of  the  (old)  ECG  cart  that  had  not  been  resolved  despite 
repeated  efforts  by  the  repairmen.  Therefore,  during  the  Y  Period  data 
collection,  operation  of  the  ECG  process  at  North  Island  was  unchanged 
from  the  baseline.  Table  4-15  summarizes  data  concerning  ECG  types  and 
readers  for  this  site. 

Table  4-16  summarizes  ECG  service  effectiveness  with  manual  services 
(X  Period  and  currently)  and  the  projected  values  when  the  system  is 
fully  implemented.  With  non-automated  services,  all  non-emergency  ECGs 
were  sent  to  San  Diego  NRMC  to  be  read.  Emergency  ECGs  were  taken  at 
North  Island  and  then  the  patient  with  his  ECG  was  sent  to  San  Diego 
NRMC  where  the  ECG  was  read  and  the  patient  evaluated.  Using  this  process. 
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TABLE  4-15 


SUMMARY  OF  QUANTITATIVE  DATA  FOR 
NAVAL  AIR  STATION  NORTH  ISLAND  BRANCH  CLINIC 


Volume 


X  Period  Projected 

3,149  3,149 


IZ£e 

Routine  Outpatient  20%  20% 

Annual  Physical  75%  75% 

STAT/ER  5%  5% 

Reader 

Cardiologist  95%  10% 

Internist  0  0 

Other  5%  90% 

Interpretation 

Normal  90%  90% 

Abnormal  10%  10% 
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*Not  implemented  during  Y  Period  data  collection. 


70  percent  of  all  ECGs  met  effective  service  criteria. 

When  CAPOC  becomes  operational  at  North  Island,  the  percent  of  ECGs 
meeting  effective  service  criteria  is  expected  to  increase  to  98  percent 
This  requires  that  all  ECGs  be  captured  by  CAPOC  and  that  80  percent  of 
abnormal  ECGs  receive  a  serial  comparison  when  the  automated  storage  and 
retrieval  capability  is  fully  operational. 

Conclusions :  The  CAPOC  system  was  not  operational  at  North  Island  in 
the  fall  of  1980  due  to  reliability  problems  with  their  old  ECG  cart. 
Service  effectiveness  is  expected  to  increase  with  automation  when  this 
problem  is  solved  and  when  automatic  storage  and  retrieval  of  ECGs  is 
available  on  the  CAPOC  system. 
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I. 


NAVAL  TRAINING  CENTER  BRANCH  CLINIC 


The  Naval  Training  Center  Branch  Clinic  (NTC)  is  an  outpatient 
clinic  located  in  San  Diego.  There  were  six  physicians  at  NTC — one  in¬ 
ternist  and  five  general  medical  officers — during  the  X  Period.  The 
internist  left  between  the  X  and  Y  Period  site  visits  so  staffing  was 
reduced  to  five  general  medical  officers  in  the  Y  Period.  Specialty 
clinics  are  staffed  by  physicians  from  San  Diego  NRMC.  Patients  requir¬ 
ing  specialized  care  and  services  are  referred  to  San  Diego  NRMC. 

In  the  X  and  Y  Periods  organization  of  the  ECG  process  was  similar. 

The  ECG  Laboratory  was  located  in  the  Physical  Examination  Clinic.  ECGs 
were  taken  from  0730  to  1600  on  Monday  through  Friday.  ECGs  were  avail¬ 
able  in  the  emergency  room  at  night  and  on  weekends.  However,  most 
patients  with  cardiac  symptoms  were  sent  to  the  emergency  room  at  San 
Diego  NRMC. 

The  ECG  Laboratory  was  staffed  by  one  full-time  technician  during 
the  X  Period.  This  person  did  not,  however,  devote  full-time  to  ECGs. 

In  the  Y  Period,  approximately  10  corpsmen  were  responsible  for  taking 
ECGs  on  a  part-time,  rotating  basis.  The  ECG  equipment  at  NTC  did  not 
change  as  a  result  of  CAPOC  implementation.  NTC  purchased  a  three-chan¬ 
nel  transmitting  ECG  cart  in  1977  in  anticipation  of  CAPOC.  A  three- 
channel,  non-transmitting  ECG  cart,  purchased  in  1975,  was  kept  for  back¬ 
up  use.  The  CAPOC  printer  is  located  in  the  ECG  Laboratory. 

The  annual  ECG  volume  as  measured  by  logs  kept  at  the  facility  in  the 
X  Period  was  1307  ECGs  (Table  4-17).  This  increased  by  29  percent  in  the  Y 
Period  to  1679  ECGs.  However,  only  31  percent  of  these  ECGs  were  being 
processed  by  CAPOC  in  the  fall  of  1980  when  Y  Period  data  collection 
occurred.  A  lack  of  training  and  the  high  turnover  rate  of  technician 
staff  were  given  as  reasons  for  this  low  level  of  system  utilization. 

Tables  4-17  and  4-18  summarize  basic  data  on  ECG  types,  turnaround 
time  and  ECG  reading  and  the  impacts  of  CAPOC  on  ECG  services  at  NTC.  In 
the  X  Period  12  percent  of  the  ECGs  met  effective  service  criteria.  This 
represents  the  ECGs  read  by  the  internist.  In  the  Y  Period  24  percent 
of  the  ECGs  met  effective  service  criteria.  This  change  is  due  to  the 
general  medical  officers  becoming  preferred  readers  for  those  ECGs 
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TABLE  4-17 


SUMMARY  OF  QUANTITATIVE  DATA  FOR 
NAVAL  TRAINING  CENTER  BRANCH  CLINIC 


X  Period  Y  Period 

Volume  1,302  1,679 

TyPe 

Routine  Outpatient  20%  20% 

Annual  Physical  80%  80% 

Reader 

Cardiologist  0  0 

Internist  12%  7.4% 

Other  88%  92.6% 

Interpretation 

Normal  95%  76% 

Abnormal  5%  24% 
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T0TAL  156  (12.0)  396  (23.6)  1,598  (95.2) 


processed  by  CAPOC.  It  is  projected  that,  with  full  implementation  of 
CAPOC,  9b  percent  of  all  ECGs  will  meet  effective  service  criteria.  This 
projection  is  based  on  two  assumptions:  all  ECGs  done  at  NTC  will  be 
processed  by  CAPOC  and  80  percent  of  abnorraals  will  receive  a  serial 
comparison  when  the  automatic  storage  and  retrieval  capability  is  operating. 

The  NTC  staff  indicated  that  they  are  satisfied  with  CAPOC,  when 
it  is  used.  They  especially  liked  the  rapid  turnaround  time  for  uncon¬ 
firmed  interpretations.  They  wculd  like  to  see  more  frequent  training 
of  technicians.  (A  change  from  part-time  back  to  a  full-time  ECG  tech¬ 
nician  might  also  help  in  solving  the  problem  of  low  utilization  due  to 
lack  of  technician  training  in  system  use.) 

Conclusions :  The  overall  effectiveness  of  ECG  services  as  measured  by 
the  criteria  increased  despite  the  internist  leaving  and  only  31  percent 
of  the  ECGs  being  processed  by  CAPOC.  Service  effectiveness  should  in¬ 
crease  even  more  upon  full  implementation.  A  more  frequent  training 
program  is  needed  to  insure  that  all  technicians  can  use  the  CAPOC 

system,  and,  thus,  that  all  ECGs  taken  at  this  site  will  be  processed  by 
CAPOC. 


174 


Arthur  D  little  Inc 


J.  NAVAL  AMPHIBIOUS  BASE  CORONADO  BRANCH  CLINIC 


The  Naval  Amphibious  Base  Coronado  Branch  Clinic  (Coronado)  is  an 
outpatient  dispensary  located  about  four  miles  from  San  Diego  across  the 
San  Diego-Coronado  Bay  Bridge.  The  population  served  is  predominantly 
dependents  of  active-duty  military  and  retired  personnel.  The  facility 
was  staffed  by  nine  general  practitioners  in  both  the  X  and  Y  data  col¬ 
lection  periods.  Several  specialty  clinics,  staffed  on  a  part-time 
basis  by  physicians  from  San  Diego  NRMC,  are  provided  at  Coronado. 

Patients  with  special  problems  or  in  need  of  hospitalization  are  re¬ 
ferred  to  San  Diego  NRMC.  Outpatient  visits  totaled  67,000  in  the  X 
Period  and  73,000  in  the  Y  Period,  an  increase  of  9  percent. 

The  process  for  taking  ECGs  at  Coronado  was  the  same  in  the  X  and 
Y  data  collection  periods.  The  ECG  Laboratory,  open  from  0800  to  1500 
on  Monday  through  Friday,  was  staffed  by  one  full-time  technician  and 
one  additional  corpsman  who  spent  about  20  percent  of  the  time  taking 
ECGs.  ECGs  were  done  in  the  Emergency  Room  at  night  and  on  weekends. 

ECG  equipment  was  also  the  same  before  and  after  CAPOC  implementation. 

A  three-channel  transmitting  cart  was  purchased  in  1978  in  anticipation 
of  CAPOC.  A  single-channel  ECG  cart,  purchased  in  1972,  was  kept  for 
back-up  use.  The  CAPOC  printer  is  located  in  the  ECG  Laboratory. 

The  process  of  reading  ECGs  taken  at  Coronado  changed  between  the 
X  and  Y  Periods.  In  the  X  Period,  ECGs  taken  as  part  of  annual  physical 
examinations  were  sent  to  San  Diego  NRMC  to  be  read  by  the  cardiology 
staff.  STAT  ECGs  and  ECGs  for  routine  outpatient  visits  were  read  by 
the  ordering  physician.  In  the  Y  Period,  all  ECGs  were  read  by  a  car¬ 
diologist  at  San  Diego  NRMC.  (Emergency  Room  ECGs  were  also  read  imme¬ 
diately  by  the  requesting  physician.) 

Table  4-19  presents  data  on  ECG  volumes  and  types  and  ECG  reading. 

In  the  X  Period,  the  ECG  volume  was  1,200.  This  volume  increased  by  5  per¬ 
cent  in  the  Y  Period  to  1,307  ECGs.  This  increase  is  consistent  with  the 
increase  in  outpatient  visits  that  occurred  during  the  same  time  period. 

Table  4-20  summarizes  the  measured  and  projected  levels  of  effec¬ 
tive  ECG  services  at  Coronado.  In  the  X  Period,  9  percent  of  all  ECGs 
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TABLE  4-19 


SUMMARY  OF  QUANTITATIVE  DATA  FOR 
NAVAL  AMPHIBIOUS  BASE  CORONADO  BRANCH  CLINIC 


X  Period  Y  Period 

Volume  1,200  1,307 

Type 

Routine  Outpatient  48%  59% 

Annual  Physical  50%  39% 

STAT/ER  2%  2% 

Reader 

Cardiologist  50%  31.4% 

Internist  0  0 

Other  50%  68.6% 


Interpretation 

Normal  92%  66.3% 

Abnormal  8%  33.7% 
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TOTAL  108  (9.0)  886  (67.8)  1,214  (92.9) 


met  effective  service  criteria  before  CAPOC  was  installed.  This  had 
increased  to  68  percent  in  the  Y  Period  in  the  fall  of  1980  after  CAPOC 
had  been  in  use  for  9  months.  The  increase  can  be  attributed  to  the 
general  practitioners  being  classified  as  preferred  readers  for  confirming 
computer-generated  normal  ECG  interpretations  and  a  decrease  in  turn¬ 
around  time  for  the  routine  ECGs  (those  not  done  in  the  Emergency  Room). 

In  the  Y  data  collection  period,  use  of  serial  comparison  had  dropped 
because  the  system's  automatic  storage  and  retrieval  capability  was 
not  in  operation.  Only  Emergency  Room  ECGs  had  a  serial  comparison. 

At  full  implementation  of  CAPOC,  it  is  projected  that  93  percent  of  all 
ECGs  at  Coronado  will  meet  effective  service  criteria.  This  change  will 
occur  solely  from  80  percent  of  non-emergency  ECGs  that  are  abnormal 
receiving  a  serial  comparison  when  the  automatic  storage  and  retrieval 
feature  is  fully  operational. 

Physicians  at  Coronado  reported  that  they  like  the  CAPOC  system, 
especially  the  rapid  turnaround  time  for  return  of  unconfirmed  inter¬ 
pretations.  Also,  the  typed,  confirmed  interpretations  were  more  legible 
than  the  handwritten  reports  received  in  the  X  Period.  The  ECG  techni¬ 
cian  also  liked  the  CAPOC  system.  She  believed  that  CAPOC  provided 
better  organization  and  resulted  in  fewer  lost  ECGs.  She  did  not  be¬ 
lieve  it  had  decreased  total  time  devoted  to  the  ECG  process  by  techni¬ 
cians. 

Conclusions :  CAPOC  has  resulted  in  a  dramatic  increase  in  service 

effectiveness.  The  general  practitioners  became  preferred  readers  for 
normal  ECGs  and  the  turnaround  time  decreased  for  ECGs  done  as  part  of 
annual  physicals  examinations. 
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K.  MARINE  CORPS  RECRUIT  DEPOT  BRANCH  CLINIC 


The  Marine  Corps  Recruit  Depot  (MCRD)  Branch  Clinic  is  an  out¬ 
patient  facility  on  a  recruit  training  base  in  San  Diego.  Outpatient 
visits  were  approximately  129,000  in  the  X  Period  and  dropped  to  slightly 
over  124,000  in  the  Y  Period.  The  medical  staff  consisted  of  four  full¬ 
time  general  practitioners  in  both  periods.  Specialists  from  San  Diego 
NRMC  staffed  the  specialty  clinics  on  a  part-time  basis. 

Both  before  and  after  CAP0C  was  installed,  the  ECG  Laboratory,  lo¬ 
cated  in  the  General  Practice  Clinic,  was  staffed  by  nine  part-time  tech¬ 
nicians.  ECGs  were  taken  Monday  through  Friday  from  0730  to  1600.  There 
was  a  corpsman  on  duty  to  take  ECGs  during  nights  and  weekends. 

A  three-channel  Marquette  3000  transmitting  cart  was  purchased  in 
1976  in  anticipation  of  CAPOC.  A  single-channel  ECG  cart,  purchased  in 
1977,  was  kept  for  back-up  use.  A  printer,  for  use  with  CAPOC  was  lo¬ 
cated  in  the  ECG  Laboratory. 

Basic  data  on  E^G  volumes  and  types  and  specialized  t  aining  of  ECG 
readers  are  summarized  in  Table  -21.  In  the  X  Period  the  annual  ECG  volume 
at  MCRD  was  900.  This  number  decreased  to  750  in  the  Y  Period.  Some  of 
this  change  might  be  related  to  the  drop  in  outpatient  visits;  the  change 
may  also  be  due  to  more  accurate  reporting  by  the  CAPOC  system. 

The  impacts  of  CAPOC  on  MCRD  for  the  X  Period,  Y  Period  and  full 
implementation  are  given  in  Table  4-22.  Before  automation,  physicians 
at  MCRD  read  all  their  own  ECGs.  Since  these  readers  did  not  meet  the 
criterion  for  effective  readers,  no  ECGs  met  effective  service  criteria 
during  the  X  Period.  With  CAPOC,  however,  ECGs  were  being  read  by  the 
cardiologists  at  San  Diego  NRMC  and  unconfirmed  interpretations  were 
available  to  the  physicians  at  MCRD.  The  general  practitioners  at  MCRD 
met  preferred  criteria  for  ECG  reader  when  confirming  normal  interpretations 
generated  by  CAPOC.  These  ECGs  were  also  available  in  a  preferred  turn¬ 
around  time.  The  cardiologists  at  San  Diego  NRMC  were  preferred  readers 
for  abnormal  ECGs.  These  were  available  within  an  acceptable  time  for 
routine  outpatients  and  within  a  preferred  time  for  ECGs  done  as  part  of 

annual  physicals.  However,  no  serial  comparisons  were  available  during 
the  Y  Period  data  collection  because  the  automated  storage  and  retrieval 


179 


3/2 


^D-fti4i  007  EVALUATION  OF  AUTOMATED  ECG  (ELECTROCARDIOGRAMS) 

SERVICES:  A  STUDV  OF  THE..  (U)  LITTLE  (ARTHUR  D)  INC 
CAMBRIDGE  MA  21  JUL  81  MDA902-88-C-0484 
UNCLASSIFIED  F/G  6/5 


NL 


I 


MICROCOPY  RESOLUTION  TEST  CHART 

NATIONAL  BUREAU  OF  STANDARDS-1963-A 


TABLE  4-21 


SUMMARY  OF  QUANTITATIVE  DATA  FOR 
MARINE  CORPS  RECRUIT  DEPOT  BRANCH  CLINIC 


Volume 


X  Period 


Y  Period 


900 


750 


Type 

Routine  Outpatient  48%  46% 

Annual  Physical  47%  52% 

STAT/ER  5%  2% 

Reader 

Cardiologist  0  28.8% 

Internist  0  0 

Other  100%  71.2% 

Interpretation 

Normal  95%  70.1% 

Abnormal  5%  29.9% 
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TOTAL 


capabilities  of  the  system  were  not  operational.  Changes  introduced 
by  CAPOC  resulted  in  70  percent  of  all  ECGs  meeting  effective  service 
criteria  in  the  Y  Period.  When  automated  storage  and  retrieval  for 
serial  comparison  becomes  operational,  it  is  estimated  that  80  percent 
of  abnormal  ECGs  will  receive  a  serial  comparison.  This  change  will 
mean  that  93  percent  of  all  ECGs  at  MCRD  will  meet  effective  service 
criteria  upon  full  implementation  of  CAPOC. 

Physicians  at  MCRD  reported  that  they  liked  the  CAPOC  system,  es¬ 
pecially  the  rapid  turnaround  time  for  the  unconfirmed  interpretation. 
Their  reliance  on  the  unconfirmed  interpretation  was  increasing  as  they 
became  more  experienced  with  the  system. 

The  only  problem  identified  during  the  Y  Period  site  visit  was 
with  training  of  corpsmen.  Some  of  this  problem  was  due  to  the  high 
turnover  rate  of  corpamen  at  the  facility  and  the  infrequent  schedule 
of  formal  CAPOC  training  programs. 

Conclusions :  CAPOC  has  resulted  in  improvements  in  ECG  services  at  the 
Marine  Corps  Recruit  Depot.  Seventy-two  percent  of  all  ECGs  met  criteria 
for  effective  ECG  services  whereas,  in  a  manual  mode,  no  ECGs  met  the 
criteria.  When  automated  storage  and  retrieval  of  old  tracings  is 
available,  93  percent  of  all  the  ECGs  at  MCRD  will  meet  criteria  for 
effective  services. 
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.  PORT  HUENEME  NAVAL  REGIONAL  MEDICAL  CLINIC 


Fort  Hueneme  is  located  approximately  50  miles  northwest  of  Los 
Angeles  and  just  south  of  Oxnard,  California.  The  Naval  Regional  Medi¬ 
cal  Clinic  at  this  facility  provides  outpatient  medical  care  only. 
Emergency  cases  are  sent  to  the  nearby  local  civilian  hospital.  Other 
patients  requiring  specialized  care  are  referred  to  Long  Beach  Naval 
Regional  Medical  Center.  At  the  time  of  the  Y  Period  site  visit,  there 
were  11  physicians  on  the  staff — one  internist,  eight  general  medical 
officers,  and  two  pediatricians.  One  internist  and  the  flight  surgeon 
left  and  one  general  medical  officer  was  assigned  to  the  clinic  between 
the  X  and  Y  Period  site  visits. 

The  ECG  Laboratory  was  located  in  the  Internal  Medicine  Clinic. 

In  the  X  Period,  responsibilities  for  ECGs  were  rotated  among  the  nurses 
and  corpsmen  at  the  clinic,  on  a  two-week  basis.  This  arrangement 
changed  in  the  Y  Period  such  that  a  civilian  RN  was  responsible  for  the 
Laboratory.  In  addition,  during  the  Y  Period,  a  corpsman,  assigned  to 
duty  in  the  Internal  Medicine  Clinic,  could  also  take  ECGs.  With  manual 
services,  ECG  interpretations  were  typed  after  being  dictated  by  the  in¬ 
ternist.  With  CAPOC,  this  activity  had  been  eliminated.  The  CAPOC  in- 
terprepation  was  attached  to  the  ECG.  Prior  ECGs  continued  to  be  filed  in 
the  Internal  Medicine  Clinic  after  CAPOC  implementation. 

The  clinic  had  a  three-channel  transmitting  Hewlett-Packard  ECG 
cart  in  the  X  Period,  and  a  printer  was  added  to  the  ECG  Laboratory  in 
the  Y  period.  The  ECG  cart  was  purchased  in  1977  in  anticipation  of 
CAPOC.  No  additional  equipment  at  the  site  could  be  interfaced 
with  CAPOC.  There  were  two  single-channel  ECG  machines — one  in  the 
Internal  Medicine  Clinic,  and  one  in  the  treatment  area.  In  both  periods, 
these  were  for  use  by  reservists  on  weekends.  ECGs  were  taken  between 
0730  and  1600,  Monday  through  Friday,  in  both  periods.  Patients  are 
referred  to  the  local  hospital  when  an  ECG  is  required  outside  these 
hours. 

The  annual  volume  of  ECGs  done  at  Port  Hueneme  for  fiscal  year  1979 
was  1699;  it  decreased  in  fiscal  year  1980  to  1313  ECGs.  (Table  4-23) 
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This  decrease  is  attributed  to  a  decrease  in  physician  staff,  and  the 
change  to  a  clinic  from  a  hospital  (which  closed  in  June,  1979). 

A  summary  of  the  basic  data  and  impacts  of  CAPOC  on  Port  Hueneme 
Naval  Regional  Medical  Clinic  are  shown  in  Tables  4-23  and  4-24.  During 
X  Period  data  collection,  no  data  were  available  on  types  of  ECGs,  serial 
comparison,  etc. ;  rather,  interviews  were  relied  upon  for  necessary 
breakdowns.  In  the  Y  Period  data  collection,  logs  were  available  that 
contained  the  necessary  information.  Using  data  on  serial  comparisons, 
percent  normals  and  types  of  ECGs,  85.5  percent  of  all  ECGs  met  effective 
service  criteria  in  X  and  Y.  However,  the  split  between  preferred  and 
acceptable  ECGs  did  change.  The  internist  became  a  preferred  reader  and 
turnaround  time  improved  because  ECG  reports  were  no  longer  delayed  by 
typing.  This  resulted  in  13.1  percent  of  ECGs  meeting  criteria  for 
preferred  services.  Breakdowns  on  ECG  types  and  serial  comparisons  col¬ 
lected  in  the  Y  Period  were  applied  to  the  X  Period.  It  was  felt  that 
these  data  were  more  reliable  than  estimates  provided  by  interviews  with 
various  personnel  at  the  facility.  However,  if  the  staff  estimates  were 
correct  in  the  X  Period,  92  percent  of  all  ECGs  would  have  met  effective 
service  criteria. 

At  full  implementation,  it  is  projected  that  93.5  percent  of  all 
ECGs  will  meet  effective  criteria.  This  would  require  that  the  patient 
be  given  a  copy  of  the  unconfirmed  interpretation  to  take  back  to  the 
requesting  physician.  It  would  also  require  that  all  abnormal  ECGs  be 
read  by  the  cardiologist  at  Long  Beach  Naval  Regional  Medical  Center  and 
80  percent  of  these  receive  a  serial  comparison. 

The  staff  at  Port  Hueneme  NRMC  likes  the  CAPOC  system.  They  report 
that  it  gives  them  confidence  in  reading  and  serves  to  make  them  more 
aware  of  various  subtle  points  that  they  may  miss  in  reading.  The 
internist  believes  it  reduces  reading  time  for  him. 

There  is  some  problem  with  the  operational  aspects  of  the  system. 
Often  the  printer  is  turned  off  and  the  central  system  continually  dials 
the  site,  but  the  unconfirmed  interpretation  is  not  received.  This  prob¬ 
lem,  attributed  to  a  lack  of  technician  training,  will  hopefully  be  re¬ 
solved  when  the  CAPOC  training  program  is  fully  operational. 
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TABLE  4-23 

SUMMARY  OF  QUANTITATIVE  DATA  FOR 
PORT  HUENEME  NAVAL  REGIONAL  MEDICAL  CLINIC 


Type 


Reader 


Interpretation 


X  Period 


Y  Period 


Volume 


1,699 


1,313 


Routine  Outpatient 
Annual  Physical 


82. 5% 
17.5% 


82.5 % 
17.5% 


Cardiologist 

Internist 

Other 


26.9% 

67.6% 


Normal 

Abnormal 


67.6% 

32.4% 


67.6% 

32.4% 
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Conclusions :  Overall,  ECG  service  effectiveness  did  not  change  at  Port 
Hueneme  after  the  implementation  of  CAPOC.  However,  some  of  the  ECGs 
meeting  acceptable  service  criteria  in  the  X  Period  met  preferred  service 
criteria  in  the  Y  Period.  Full  implementation  is  expected  to  increase 
the  overall  service  effectiveness. 
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M.  BARSTOW  BRANCH  CLINIC 

Barstow  Branch  Clinic  is  an  outpatient  facility  located  about  70 
miles  north  of  San  Bernardino.  This  clinic  provides  general  outpatient 
medical  services.  Patients  requiring  diagnostic  or  therapeutic  services 
that  are  not  available  at  Barstow  are  referred  to  Camp  Pendleton  Naval 
Regional  Medical  Center  or,  in  emergencies,  to  local  civilian  hospitals. 

The  staff  consists  of  four  general  practitioners. 

Before  CAPOC  was  installed,  the  ECG  Laboratory  was  located  in  a 
small  room  adjacent  to  the  Emergency  Room.  It  was  staffed  by  seven 
part-time  technicians.  ECGs  were  done  on  Monday  through  Friday  from 
0700  to  1530  hours.  On  nights  and  weekends,  ECGs  were  taken  by  the 
corpsman  assigned  to  the  ER.  There  was  a  three-channel  Hewlett-Packard, 
Model  1517A,  transmitting  cart  located  in  the  ECG  room  adjacent  to  the 
ER.  In  addition,  there  was  a  one-channel  machine  kept  for  use  as  a 
back-up . 

CAPOC  had  not  been  implemented  at  this  site  during  the  Y  Period 
site  visits.  They  were  still  experiencing  difficulties  in  transmitting 
ECGs  and  with  the  operation  of  their  ECG  cart.  However,  the  ECG  process 
is  not  expected  to  change  significantly  when  CAPOC  is  implemented. 

For  the  year  October,  1978,  to  September,  1979,  approximately  860 
ECGs  were  taken  at  Barstow  (Table  4-25).  This  figure  was  estimated  based 
upon  the  log  book  and  interviews  with  staff.  According  to  estimates  of 
the  physician  staff,  about  45  percent  of  the  ECGs  were  taken  for  routine 
outpatient  examinations,  50  percent  as  part  of  annual  physical  examina¬ 
tions,  and  5  percent  for  emergency  patients.  It  is  assumed  that  these 
breakdowns  and  volumes  will  not  change  when  CAPOC  is  eventually  implemented. 

Summaries  of  basic  data  and  the  projected  Impacts  of  CAPOC  compared 
with  X  Period  manual  services  are  given  in  Tables  4-25  and  4-26.  No  ECGs 
met  effective  service  criteria  when  using  non-automated  services.  When 
full  Implementation  occurs,  it  is  estimated  that  almost  100  percent  of 
all  ECGs  will  meet  effective  service  criteria.  The  increase  will  result 
from  two  changes.  The  general  medical  officers  will  become  preferred 
readers  for  all  normal  ECGs,  estimated  at  99  percent  of  ,;he  volume. 
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TABLE  4-25 


SUMMARY  OF  QUANTITATIVE  DATA  FOR 
BARSTOW  BRANCH  CLINIC 


X  Period 


Projected 


Volume 

lyPg. 

Routine  Outpatient 
Annual  Physical 
Pre-Operative 

Reader 


Cardiologist 

Internist 

Other 


99.0% 


Interpretation 


Normal 


Abnormal 
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*Not  implemented  during  Y  Period  data  collection. 


•l 


Cardiologists  at  Camp  Pendleton  will  provide  interpretations  for  abnormal 
ECGs  as  well  as  serial  comparisons  for  80  percent  of  abnormal  ECGs.  (This 
assumes  that  the  automated  storage  and  retrieval  capability  is  available.) 

Conclusions :  When  CAPOC  services  are  available,  they  will  result  in  an 
increase  of  the  effectiveness  of  ECG  services  at  Barstow  Branch  Clinic. 


CHAPTER  5.  RESULTS 


A.  IMPACT  OF  CAPOC  ON  EFFECTIVENESS  OF  ECG  SERVICE 

The  primary  objective  in  installing  the  CAPOC  system  is  to  improve 
ECG  services  in  the  18  CAPOC  sites  in  the  region.  To  be  useful,  an  ECG 
interpretation  must  be  accurate,  and  must  be  available  when  needed  for 
the  diagnosis  and  treatment  of  the  patient.  For  this  evaluation,  an 
effective  ECG  was  defined  as  one  that  is  read  by  a  qualified  reader,  is 
returned  to  the  requesting  physician  within  a  prescribed  time,  and  is 
compared  with  a  prior  interpretation  if  the  current  ECG  is  abnormal.  As 
described  in  Chapter  2,  the  required  turnaround  time  depends  on  the  type 
of  patient  being  treated,  and  hence,  the  way  the  ECG  is  used.  ECGs  for 
emergency  patients  are  needed  within  minutes  so  that  the  physician 
can  reflect  the  results  in  immediate  decisions  that  have  to  be  made  con¬ 
cerning  patient  management,  whereas  screening  ECGs  taken  as  part  of  annual 
physical  examinations  are  required  within  a  few  days  of  the  examination. 

Two  effectiveness  parameters — reading  quality  and  availability  of  the 
interpretation — were  identified  as  the  most  important  service  criteria  by 
potential  users  in  the  region,  and  by  the  review  panel.  (Reading  quality 
includes  both  the  training  of  the  reader  and  use  of  serial  comparison  for 
abnormal  ECGs.)  Therefore,  these  parameters  were  the  focus  of  data 
collection  and  are  discussed  in  detail  here.  Other  aspects  of  high-quality 
ECG  services  were  evaluated  in  a  qualitative  manner  and  those  findings  are 
discussed  in  Section  D  of  this  chapter. 

In  order  to  evaluate  the  key  service  criteria  of  reader,  turnaround 
time,  and  use  of  serial  comparison,  two  levels  of  service  were  defined— 
acceptable  service  and  preferred  service.  These  two  levels  together 
comprise  effective  services.  The  criteria  for  each  service  level  are 
discussed  in  detail  in  Chapter  2.  The  percentage  of  ECGs  meeting  each 
service  level  were  determined  under  manual  operations  (X  Period)  in  the 
Fall  of  1979;  in  the  Fall  of  1980  when  the  system  was  partially  opera¬ 
tional  (Y  Period);  and  then  were  projected  for  the  system  when  effective¬ 
ness  has  been  maximized  by  resolution  of  remaining  technical  problems  and 
adoption  of  operational  changes  at  some  sites  (full  implementation). 
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The  results  of  the  analysis  of  ECGs  meeting  effective  service 
criteria  are  shown  in  Table  5-1,  which  lists  the  18  sites  in  order  of  de¬ 
creasing  ECG  volume,  and  grouped  into  two  categories,  large  sites  and 
small  sites.  Large  sites  are  those  sites  with  more  than  100  beds  and/or 
more  than  200,000  outpatient  visits  per  year. 

The  table  presents  data  on  the  percentage  of  the  ECGs  at  each  site 

that  met  effective  service  criteria  in  the  X  and  Y  evaluation  periods,  and 

the  percentage  of  ECGs  that  are  projected  to  meet  the  criteria  when 

CAPOC  has  been  fully  implemented.  Before  the  implementation  of  CAPOC, 

50%  of  all  ECG  services  in  the  region  met  the  criteria  for  effective 

ECG  services:  56  percent  of  the  ECGs  in  the  large  sites  and  38  percent 

12 

of  those  in  small  sites. 

The  table  also  shows  the  service  elements  that  inhibited  achieve- 
ment  of  effective  service  levels  in  each  site.  In  large  facilities, 
turnaround  time  to  receive  the  interpretation  and  use  of  serial  compari¬ 
son  were  the  most  common  problem  areas.  This  is  because  large  sites 
have  a  high  percentage  of  abnormal  ECGs,  which  require  a  serial  com¬ 
parison  and  a  high  percentage  of  emergency,  pre-operative,  and  inpatient 
ECGs,  which  require  rapid  turnaround  for  an  interpretation.  When  ECGs 
were  processed  manually  at  small  sites  without  cardiologists  or  in¬ 
ternists  on  the  staff,  the  tracings  were  typically  read  by  a  general 
practitioner  because  it  took  too  long  to  receive  an  interpretation  from 
a  large  site  by  mail.  Therefore,  a  large  number  of  ECGs  at  these  sites 
did  not  meet  the  criteria  for  effective  reader.  In  small  sites  where 
ECGs  were  sent  outside  to  be  read,  services  did  not  meet  criteria  for 
effective  turnaround  time. 

When  the  CAPOC  system  is  fully  operational,  effectiveness  of  ser¬ 
vices  in  the  region  will  increase  from  50  percent  to  78  percent  over  the 
entire  region.  The  most  dramatic  increase  in  effectiveness  will  occur 


These  ECGs  were  read  by  a  qualified  reader,  were  returned  within  the 
specified  time  limits,  and  received  a  serial  comparison  if  abnormal.  An 
ECG  had  to  meet  all  three  of  these  service  criteria  in  order  to  be  con- 
sidered  effective. 
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at  small  sites,  where  effectiveness  will  more  than  double  from  38  percent 
to  86  percent. 

At  large  sites,  the  increase  in  effectiveness  of  services  after 
automation  is  primarily  due  to  the  Increased  availability  of  prior  ECG 
tracings  that  results  from  the  automated  storage  and  retrieval  of  ECGs 
by  CAPOC.  Misfiled  records  contribute  to  reduced  effectiveness  of  manual 
ECG  services.  In  one  large  site,  80  ECGs  were  Identified  for  which  the 
patient  indicated  that  a  prior  ECG  had  been  taken  at  the  site,  but  for 
which  no  ECG  file  was  found  by  the  corspman.  For  48  percent  of  these,  a 
search  through  the  files  revealed  that  there  was  indeed  an  ECG  folder  for 
the  patient  in  the  general  area  where  the  file  should  have  been  located. 
The  computer  system  will  eliminate  the  problem  of  misfiling. 

At  all  sites  the  system  will  also  decrease  turnaround  time  to  re¬ 
ceive  some  types  of  interpretations,  for  instance,  normal  ECGs  will  be 
considered  available  as  soon  as  the  requesting  physician  receives  the 
computer  report  within  minutes  of  the  time  the  tracing  is  taken. 

At  small  sites  the  most  Important  aspect  of  CAPOC  service  is  pro¬ 
viding  rapid  access  to  computer-generated  reports  for  normal  ECGs.  Be- 

13 

cause  of  the  accuracy  of  the  system  in  interpreting  normal  tracings 
a  general  practitioner,  flight  surgeon,  or  corpsman  is  considered  quali¬ 
fied  to  confirm  a  normal  computer  report.  At  small  facilities  most 
electrocardiograms  taken  are  normal;  hence,  the  percentage  of  ECGs  meeting 
criteria  for  effective  reader  and  turnaround  time  will  increase 
dramatically  after  automation.  Small  sites  linked  to  a  large  facility 
with  specialized  readers  (cardiologists  or  Internal  medicine  specialists) 
receive  the  greatest  benefit  from  automation  because  the  abnormal  ECGs 
will  have  both  effectivr  waders  and  turnaround  times. 

Figure  5-1  illus  r  ^  he  projected  increase  in  effectiveness  of 
ECGs  services  after  am  '  .  <  md  the  resulting  effectiveness  levels 

at  each  site.  Small  . .1 tr  ichout  cardiologists  on  their  staff,  which 

are  linked  to  large  hospitals  via  the  automated  system,  experience 


13 

A.  D.  Hagen  and  J.  S.  Alper,  "Evaluation  of  Computer  Programs  for 
Clinical  Electrocardiography."  Circulation  51-52,  Suppl.  11-193,  1975. 
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FIGURE  5-1  EFFECTIVENESS  OF  ECG  SERVICES  -  X  PERIOD  AND  FULL  IMPLEMENTATION 


dramatic  increases  in  service  effectiveness.  The  CAPOC  system  clearly 
meets  the  objective  of  improving  ECG  services  at  MTFs  without  a  cardiology 
staff. 


The  impact  of  the  CAPOC  system  on  service  effectiveness  is  even 
more  dramatic  when  the  percentage  of  ECGs  meeting  preferred  service 
levels — the  more  stringent  criteria — are  examined.  As  shown  in  Table  5-2, 
the  percentage  of  ECGs  in  the  region  meeting  these  criteria  before  the 
implementation  of  CAPOC  was  very  low — 7  percent.  After  CAPOC  has  been 
fully  implemented,  the  level  will  increase  to  51  percent.  The  most 
dramatic  increases  are  again  at  small  sites,  where  rapid  availability  of 
the  computer  interpretation  increases  the  effectiveness  of  the  ECG  in¬ 
terpretation  in  patient  care — especially  for  normal  electrocardiograms. 

B,  COST  OF  ECG  SERVICES 

The  cost  of  ECG  services  for  the  X  Period  and  when  CAPOC  is  fully 
Implemented  were  determined  based  on  the  volume  of  ECGs  processed  in 
each  period.  The  costs  of  ECG  services  in  the  X  Period  are  summarized 
by  site  and  cost  element  in  Table  5-3.  In  the  X  Period,  the  total  cost 
of  ECG  service  was  $592,776  or  $7.41  per  ECG  processed.  Of  this  total, 
$6.54  represented  variable  costs — labor  and  supplies — and  the  remaining 
$0.87  represented  fixed  costs  for  equipment. 

The  differential  cost  analysis  performed  to  determine  the  cost  of 
CAPOC  focused  on  two  areas: 

•  additional  costs  due  to  new  equipment  purchased  (both 

at  the  sites  and  for  the  central  system)  and  those  costs 
due  to  different  types  of  supplies  and  usage,  and 

•  cost  savings  attributable  to  reduction  in  labor  time 
devoted  to  obtaining  and  reading  ECGs  and  to  elimination 
of  various  types  of  supplies. 

Not  all  cos ts  or  savings  are  realized  at  each  site  due  to  the  manner 
in  which  the  ECG  process  and  overreading  occur.  The  costs  and  savings 
were  only  attributed  to  those  sites  where  they  are  expected  to  occur. 

Thus,  the  estimate  of  differential  costs  is  a  conservative  one.  Tables 
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•Percentage  of  ECGe  meeting  criteria. 

X  -  Refers  to  X  Period  when  manual  ECG  services  ware  In  use 
X  -  Refers  to  Y  Period,  one  year  following  Installation  of  CAP0C  system 

Pull  -  Refers  to  projections  of  effectiveness  when  changes  have  been  made  which  will  result  In  achieving  maximum  benefits  from 
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Actual  costs  as  obtained  from  facility  FOOTNOTES :  *  Maintenance  and  repair  cost; 

Costs  derived  from  actual  data  already  accounted  for  in  Sai 

Estimated  costs  from  information  obtained  Diego  NRMC's  costs 

from  similar  facilities 


5-4  through  5-7  summarize  the  various  differential  costs  for  the  CAPOC 
system.  The  following  section  discusses  each  cost  element. 

As  shown  in  Table  5-4,  the  time  to  read  an  ECG  with  the  assistance  of 
the  unconfirmed  computer  interpretation  was  found  to  decrease  with  CAPOC. 
This  conclusion  is  based  on  observations  of  cardiologists  reading  ECGs 
at  San  Diego  NRMC  during  the  X  and  Y  Period  data  collection  effort.  The 
reduction  in  time  was  found  to  be  1.6  minutes/ECG  over  the  manual  system. 

The  savings  in  reading  costs  are  expected  to  occur  primarily  at  the  four 
largest  sites  in  the  area — San  Diego  NRMC,  Camp  Pendleton,  March,  and 
Long  Beach.  In  addition,  reading  cost  savings  also  occur  for  smaller 
sites  who  were  having  the  ECGs  read  at  a  major  site  under  the  manual 
system.  Savings  in  reading  time  in  the  entire  region  is  estimated  to 
affect  50,833  ECGs  processed  by  CAPOC,  or  67  percent  of  ECGs  taken  in 
the  18  MTFs. 

The  other  aspect  of  labor  savings  is  in  technician  time  to  perform 
various  portions  of  the  ECG  process.  Technician  time  is  saved  due  to  the 
elimination  of  such  tasks  as  mounting  of  single-channel  ECGs,  typing  the 
interpretation,  matching  tracings  and  interpretations,  xeroxing  interpreta¬ 
tions  and  obtaining  two  ECG  tracings  (see  Table  5-4).  The  average  over¬ 
all  technician  savings,  measured  by  timed  observations  at  San  Diego  NRMC, 
was  found  to  be  3.6  minutes/ECG.  This  savings  is  only  expected  to 
affect  the  41,312  ECGs  taken  at  the  three  sites  where  ECG  overreading 
services  are  provided — San  Diego  NRMC,  Camp  Pendleton  and  Long  Beach. 

A  reduction  of  1. 33  minutes/ECG  is  expected  to  occur  at  seven  sites  at 
which  ECGs  were  previously  mounted.  A  savings  of  2  minutes/ECG  will  be 
realized  at  one  site  where  interpretations  were  typed  under  the  manual 
system. 

The  savings  in  supply  costs  occur  due  to  the  elimination  of  four 
elements: 

•  xeroxing  ($0. 5/copy) 

•  mounts  for  single-channel  tracings  ($0. 15/mount) 

•  duplicate  tracings  —  one  for  the  patient's  medical 
record  and  one  to  be  read  ($0.04/tracing) ,  and 

•  cover  sheets  ($0. 06/sheet) . 
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TABLE  5-4 


COST  SAVINGS  FOR  CAPOC  ECG  SERVICES 

AT  FULL 

IMPLEMENTATION- 

-YEAR  1 

> 

Number  of 

ECG 

#  •* 

Sites 

Vol  ume 

Savings* 

Cost  Element 

Affected 

Affected 

per  ECG 

Savings* 

% 

Reading  Cost 

6 

50,833 

$.784 

$39,853 

Technician  Labor  Costs 

■ 

Overall 

3 

41,312 

$.422 

$17,434 

Mounting  Tracing 

7 

14,311 

$.156 

$  2,233 

•jm 

Typing  Interpretation 

1 

1,313 

$.235 

$  308 

Supplies 

s 

Xerox  Copies 

5 

44,482 

$.05 

$  2,224 

s', 

Mounts 

7 

14,311 

$.15 

$  2,147 

•*  , 

Tracings 

2 

29,792 

$.04 

$  1,192 

Cover  Sheets 

1 

14,059 

$.06 

$  844 

TOTAL  Savings 

$66,235 

« 

*1980  dollars. 
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TABLE  5-5 


CAPOC  SYSTEM  COSTS 


Marquette  Equipment  and  Services  Contract 
A.  Initial  Costs: 

System  Installation  $  15,000 

Training  Classes  7,363 

Training  Materials  6,460 

Software  Conversion  (includes  share  of 

initial  conversion)  23,000 

Software  Modification:  Storage  and  Retrieval  963 

TOTAL 


$  52,786 


B. 


Maintenance/Software : 

Software  $ 

Central  System  (including  plotter) 

Overread  Stations  (Micromuse) 

5  at  $3,540/yr.  - 

User  Site  Printers 
16  at  $372/yr.  - 

Total  Annual  Cost  for  Maintenance  and  Software  $ 
TOTAL  Year  1  and  Year  2  Maintenance  and  Software 


7,200/yr. 

20,628/yr. 

17,700/yr. 

5 ,952/yr. 

51,480/yr. 


$102,960 


C.  Communication  Costs : 

Year  1  +  2  up  to  7,000  ECGs /month 

D.  Equipment  Lease  (prior  to  10/81) : 
Central  Equipment  (including  plotter) 
Overread  Stations — 5  at  $10,836  each 
User  Site  Printers — 16  at  $1,140  each 
Total  Annual  Lease  Costs 

TOTAL  Lease  Costs  (Year  1  +  Year  2) 


$108,864 


$  63,564/yr. 

54,180/yr. 

18,240/yr. 

135,984/yr. 

$271,968 
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TABLE  5-5 


CAPOC  SYSTEM  COSTS 
(Continued) 


E.  Equipment  Purchase  Price: 

Initial 

Central  Equipment  (including  plotter) 
Overread  Stations,  5  at  $36,875  each 
Printers,  16  at  $3,870  each 
TOTAL  Equipment  Purchase  Price 

F.  Equipment  Buyout  After  Year  2: 

Initial  Cost-(.50  lease  payments  Year  1  + 
.55  lease  payments  Year  2)  =  $462,439  - 
($67,992  +  $74,791) 

G.  Recurring  Costs  Years  3-8; 

Yearly  Maintenance/Software  Base  Cost 
$51,480/yr  Inflated  at  10%  per  year 

Year  3 

Year  4 

Year  5 

Year  6 

Year  7 

Year  8 

TOTAL  Maintenance  Years  3-8 

Yearly  Communications  Costs 

$54,432/yr  Inflated  at  10%  per  year 

Year  3 

Year  4 

Year  5 

Year  6 

Year  7 

Year  8 

TOTAL  Communications  Years  3-8 


$216,144 

184,375 

61,920 

$462,439 


$319,656 


ta 


$  56,628 
62,291 
68,520 
75,372 
82,909 
91,200 


S3 


,\ 

A 


$436,920 


$  59,875 
65,863 
72,449 
79,694 
87,663 
96,430 

$461,974 
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TABLE  5-6 


SITE  COSTS  FOR  CAPOC  SYSTEM  AT  FULL  IMPLEMENTATION 


Equipment  Purchase 
Carts 

Initial  Purchase  of  7  Carts 

Modification  of  4  Carts  to  Transmit  to  CAPOC 
at  $3,000  each 

One  Additional  Cart  on  Order 

Recommended  Additional  Purchases: 

2  Carts  at  $11,000  each 
1  Printer  at  $3,870 
TOTAL 

Average  Annual  Cost* 

Supply  Costs 
Paper 

For  Overread  Stations  ($29/3000  sheets) 

Printers  at  site  ($25.37/4,000  sheets) 

Purchase  of  Floppy  Discs  and  Replacement  of 
System  Discs  as  needed 

TOTAL 


*Based  on  a  5-year  lifetime  for  carts  and  an  8-year  lifetime 
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$  67,328 

12,000 

11,000 

22,000 

3,870 

116,198 

22,949 


1,894 

145 

2,000 

4,039 

for  printers. 
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TABLE  5-7 

SUMMARY  OF  DIFFERENTIAL  COSTS  FOR  CAPOC  ECG  SERVICES 


SAVINGS 

$/Year 

Reading 

39,853 

Labor 

19,975 

Supplies 

6,407 

TOTAL 

66,235 

COSTS 

System 

Initial  Costs 

$  52,786 

Maintenance,  Software  Year  1  and  Year  2 

102,960 

Communications  Year  1  and  Year  2 

108,864 

Equipment  Lease  Year  1  and  Year  2 

271,968 

Equipment  Buyout 

319,656 

Maintenance  Totals  Years  3-8 

436,920 

Communications  Total  Years  3-8 

461,974 

Annual  Cost  for  an  8-Year  Lifetime 

219,391 

Site 

Equipment 

22,949 

Supplies 

4,039 

TOTAL 

246,379 

Net  Differential  Cost  of  CAPOC 

$180,144 

*  Based  on  an  8  year  lifetime. 
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These  savings  are  specific  for  each  site  and  depend  on  the  ECG  process 
that  existed  in  the  manual  system  and  the  changes  that  have  occurred 
as  a  result  of  CAPOC. 

Cost  increases  due  to  CAPOC  are  incurred  on  a  regional  basis  and  at 
the  individual  sites.  The  regional  cost,  summarized  in  Table  5-5,  is  the 
cost  to  lease  the  system  for  two  years  and  then  purchase  it.  After  the 
purchase,  there  are  yearly  costs  for  maintenance,  software  and  communica¬ 
tions.  The  contract  specifies  that  these  will  be  escalated  at  10% 
annually.  The  total  regional  cost  for  CAPOC  over  its  8-year  lifetime  is 
$1,755,131,  or  $219,391  annually. 

The  site  costs  include  the  purchase  of  equipment  and  supplies  for 
use  with  CAPOC.  Seven  three-channel  transmitting  ECG  carts  were  purchased. 
One  additional  cart  is  on  order  and  two  more  carts  are  recommended  to  in¬ 
crease  capture  of  ECGs  by  CAPOC  and  an  old  cart  at  one  site  needs  to  be 
replaced.  Also,  one  additional  printer  is  recommended  to  increase  access 
to  CAPOC.  The  cost  of  this  additional  equipment  has  been  included  in 
the  Y  Period  differential  cost.  Initially  four  sites  upgraded  their  ECG 
carts  in  order  to  transmit  to  CAPOC.  The  annual  yearly  cost,  assuming 
a  five-year  lifetime  for  the  ECG  carts  and  an  8  year  lifetime  for  the 
printers,  is  $22,949. 

Supply  costs  consist  of  paper  for  the  overread  stations,  paper  for 
the  printer  and  discs  for  the  computer.  Overread  paper  is  used  at  a 
rate  of  4  sheets/ECG  processed  at  the  overread  site  itself  and  2  sheets/ 
ECG  overread  for  smaller  sites.  Printer  paper  is  estimated  to  be  one- 
half  sheet  per  report.  If  an  ECG  has  both  an  unconfirmed  and  confirmed 
report,  the  total  amount  used  is  one  sheet  per  ECG.  Differential  supply 
costs  are  $4,039  in  the  base  year  and  will  inflate  at  10%  annually  over 
the  lifetime  of  the  CAPOC  system. 

The  differential  costs  and  savings  attributable  to  CAPOC  are 
summarized  in  Table  5-7.  The  additional  costs  are  $246,379  per  year. 

These  are  offset  by  savings  of  $66,235  per  year  for  a  total  differential 
cost  of  $180,144  per  year.  This  represents  $2.39  per  ECG  taken  in  the 
Y  Period. 
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A  life-cycle  cost  analysis  was  performed  using  the  costs  and  q 

savings  discussed.  The  summary  of  the  analysis  is  presented  in  Table  5-8.  > 

The  total  discounted  annual  cost  of  the  CAPOC  system  is  $116,631  or  an 
increase  of  $1.54  per  ECG  processed  in  the  Y  Period.  v 


C.  COST-EFFECTIVENESS  OF  THE  CAPOC  SYSTEM 

The  costs  of  manual  and  CAPOC  services  when  combined  with  service 
effectiveness  yield  the  cost-effectiveness  of  ECG  services  in  the  two 
periods  for  the  Southern  California  area,  and  large  and  small  MTFs.  In 
addition  to  evaluating  the  actual  cost-effectiveness  at  the  two  service 
levels,  a  cost-effectiveness  is  calculated  based  on  annual  ECG  volumes 
of  75,000  and  100,000.  The  cost-effectiveness  for  each  period  and  under 
the  two  varying  volumes  is  given  in  Table  5-9. 

In  the  X  Period,  the  cost  of  ECG  services  was  $592,776.  Of  the 
79,976  ECGs  taken,  50.3  percent  or  40,282  ECGs  met  the  criteria  for 
effective  service.  Thus  the  cost  per  effective  ECG  with  manual  services 
was  $14.72.  The  cost  per  preferred  ECG  (5,539  ECGs  in  the  X  Period)  was 
$107.02. 

In  order  to  calculate  the  total  cost  for  CAPOC  services,  the  X  Period 
costs  were  adjusted  for  the  new  volume  of  75,529  ECGs.  If  75,529  ECGs 
were  done  with  the  manual  system,  the  total  cost  would  have  been  $563,700. 
The  total  cost  for  CAPOC  services  is  this  amount  plus  the  average  annual 
lifecycle  cost  for  CAPOC  of  $116,631  or  $680,331  in  total.  The  total 
number  of  effective  ECGs  is  projected  to  be  59,171  when  CAPOC  is  fully 
Implemented.  Thus  the  cost  per  effective  ECG  with  CAPOC  services  will 
drop  to  $11.50.  The  cost  per  preferred  ECG  (38,299  ECGs)  will  decline 
to  $17.76.  The  change  in  cost  per  effective  ECG  is  more  dramatic  in 
the  small  MTFs  than  in  the  large  ones. 

In  order  to  have  a  comparable  basis  for  comparison,  the  costs  for 
manual  and  CAPOC  ECG  services  were  calculated  based  on  annual  volumes  of 
75,000  and  100,000  ECGs.  This  calculation  presents  the  overall  effect 
of  fixed  and  variable  costs  on  the  cost-effectiveness.  The  cost  per 
effective  ECG  decreases  with  increasing  volume.  However,  the  difference 
between  costs  under  the  two  systems  also  becomes  smaller. 
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TABLE  5-9 


COST-EFFECTIVENESS  OF  MANUAL  AND  CAPOC  ECG 
SERVICES  UNDER  DIFFERENT  VOLUMES 


Total  Costs 
($/Year) 

ECG  Costs 

($/ECG) 

Effective 

Preferred 

Manual  Services 

X  Period 

Southern  California  Area 
-  79,926  ECGs 

592,776 

14.72 

107.02 

Large  MTFs  -  54,998  ECGs 

418,593 

13.59 

84.57 

Small  MTFs  -  24,978  ECGs 

174,183 

18.35 

232.45 

75,000  ECGs  per  year 

560,241 

14.85 

108.26 

100,000  ECGs  per  year 

723,741 

11.14 

104.89 

CAPOC  Services 

Full  Implementation 

Southern  California  Area 

680,331 

11.50 

17.76 

-  75,529  ECGs 

Large  MTFs  -  50,729  ECGs 

478,687 

12.58 

22.47 

Small  MTFs  -  24,800  ECGs 

201,644 

11.96 

9.45 

75,000  ECGs  per  year 

677,279 

11.53 

17.81 

100,000  ECGs  per  year 

825,623 

10.54 

16.28 
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D.  QUALITATIVE  RESULTS 

The  staff  questionnaire  and  interviews  conducted  during  site  visits 
complemented  and,  in  some  cases,  extended  the  quantitative  assessment  of 
the  impacts  of  CAPOC  in  the  18  MTFs.  The  largest  number  of  questionnaires 
was  returned  by  physicians,  nurses,  and  corpsmen  at  San  Diego  NRMC  (70  in 
the  X  Period  and  107  in  the  Y  Period).  Seventy-seven  of  the  respondents 
to  the  Y  Period  survey  had  experienced  the  previous  manual  ECG  process 
and  participated  in  the  implementation  period.  Furthermore,  San  Diego 
NRMC  is  the  largest  overreading  site  in  the  CAPOC  network,  and  30  physician 
respondents  in  the  Y  Period  were  performing  ECG  reading  functions  (in¬ 
terpreting  ECGs  for  patients  treated  by  others  or  checking  interpretations 
by  others).  San  Diego  NRMC  was  the  first  site  to  be  using  CAPOC  and  thus 
had  had  a  year  of  experience  with  CAPOC  services  when  Y  Period  surveys 
were  conducted.  Therefore,  the  experience  and  opinions  of  staff  at 
San  Diego  NRMC  were  the  most  comprehensive  qualitative  information  avail¬ 
able  on  the  impacts  of  automation  on  the  overall  ECG  process.  The 
following  discussion  of  qualitative  results  focuses  on  information 
obtained  at  San  Diego  NRMC  and  incorporates  information  from  the  other 
sites  as  available  and  appropriate. 

When  asked  about  the  aspects  of  ECG  services  most  important  to  the 
use  of  the  ECG  in  patient  care,  physicians  consistently  gave  the  highest 
ratings  to  turnaround  times  (especially  for  STAT  ECGs);  accuracy,  com¬ 
pleteness,  and  consistency  of  ECG  readings;  and  availability  of  prior 
tracings  for  serial  comparison.  When  asked  about  turnaround  times  for 
interpreted  ECGs,  most  physicians  believed  that  a  STAT  ECG  was  needed 
within  30  minutes  and  a  routine  ECG  was  needed  within  24  hours.  Thus, 
the  questionnaire  results  tend  to  affirm  the  validity  of  the  measures 
used  to  evaluate  service  quality  and  generally  confirm  the  quantitative 
results  regarding  service  effectiveness  at  San  Diego  NRMC. 

Physicians  also  rated  quality  of  tracings  and  legibility  of  results 
as  very  important.  At  San  Diego  NRMC,  satisfaction  ratings  for  turn¬ 
around  times  for  tracings  and  interpretations,  technical  quality  of 
tracings,  and  ease  of  reading  and  understanding  interpretations,  all 
improved  after  automation.  (A  decline  was  reported  in  service  quality 
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for  serial  comparisons.  This  is  not  surprising  since  automated  storage 
and  retrieval  of  ECGs  was  not  available  in  the  Y  Period  and  manual  files 
were  not  being  maintained.)  Overall  declines  were  noted  in  reported 
satisfaction  with  accuracy,  completeness,  and  consistency  of  ECG  inter¬ 
pretations  during  the  Y  Period.  This  is  not  consistent  with  responses 
to  questions  regarding  changes  in  services  resulting  from  CAPOC.  Of 
those  physicians  who  believed  they  had  sufficient  exposure  to  CAPOC  to 
comment,  33  percent  cited  improved  accuracy  and  50  percent  reported  that 
interpretations  were  more  complete  with  CAPOC.  Large  numbers  of  those 
who  commented  cited  improvements  in  technical  quality  of  tracings  (57 
percent)  and  ease  of  reading  and  understanding  interpretations. 

Physicians  at  all  sites  reported  that  their  role  in  delivery  of  ECG 
services  was  not  changed  by  CAPOC.  However,  San  Diego  physicians  reported 
decreased  time  devoted  to  reading:  an  average  of  1.5  minutes  per  normal 
ECG  vs.  1.3  minutes  with  CAPOC  and  3.9  minutes  per  abnormal  ECG  vs.  3.1 
minutes  with  CAPOC. 

ECG  readers,  who  interpret  the  largest  volumes  of  ECGs  (average 
of  15  per  week  with  12  reading  more  than  25)  were  more  likely  to  report 
time  savings  (34  percent  vs.  6  percent).  Physicians  spending  less  time 
cited  computer  measurements  of  intervals  and  rate,  legibility  of  the  CAPOC 
report,  and  less  time  for  writing  and  interpretation.  Some  of  those  who 
reported  no  time  savings  mentioned  the  lack  of  a  computer  interpretation 
in  time  to  be  useful  (this  is  consistent  with  measured  turnaround  times 
for  the  Y  Period)  or  the  need  to  check  ECGs  anyway. 

In  the  quantitative  evaluation  of  ECG  service  effectiveness,  the 
specialized  training  of  the  ECG  reader  was  used  as  the  means  of  measuring 
changes  in  reading  quality.  This  was  complemented  by  the  portion  of  the 
staff  questionnaire  concerning  the  relative  accuracy  of  manual  and 
computer-assisted  interpretations.  The  opinions  of  70  physicians  at 
San  Diego  NRMC  were  used  as  the  basis  for  assessing  this  aspect  of  ECG 
services  because  they  read  the  highest  ECG  volumes  and  had  the  most 
experience  with  CAPOC.  Of  29  physician  respondents  in  the  Y  Period  who 
performed  ECG  reading  functions,  23  believed  CAPOC  readings  were  at  least 
as  accurate  as  manual  readings  overall,  and  only  6  believed  CAPOC  was  less 


212 


Arthur  D  Little  Inc  r 


MV.  W  «LV 


accurate  overall.  Only  one  of  45  respondents  believed  that  manual  reading 
was  more  accurate  for  normal  ECGs.  For  abnormal  ECGs,  28  respondents 
believed  CAPOC  was  at  least  as  accurate  vs.  16  who  believed  manual  reading 
was  more  accurate.  Generally,  these  physicians  believed  that  CAPOC  was 
more  accurate  in  identifying  ST-T  wave  abnormalities,  hypertrophy,  and 
conduction  abnormalities  than  in  detecting  rhythm  abnormalities  and 
myocardial  infarctions.  Most  of  the  physician  readers  (24  of  34  who 
responded)  felt  that  computer  errors  tended  to  be  false-positive  errors 
(calling  a  normal  ECG  abnormal)  rather  than  false -negative  errors. 

Other  impacts  of  CAPOC  reported  by  questionnaire  respondents  in¬ 
cluded  fewer  lost  or  inadequate  tracings.  Some  physicians  believed 
that  this  had  led  to  fewer  patient  recalls  for  repeat  ECGs. 

When  asked  about  overall  satisfaction  with  CAPOC,  68  physicians  at 
San  Diego  NRMC  were  satisfied  with  the  system,  versus  10  who  were  un¬ 
satisfied.  Thus,  physician  acceptance  appears  to  be  high  despite  the 
longer  than  anticipated  implementation  period.  Specific  comments  about 
problem  areas  indicate  that  some  of  the  dissatisfaction  may  be  attributed 
to  operational  or  technical  problems  rather  than  basic  capabilities  of 
the  CAPOC  system.  Several  physicians  noted  that  CAPOC  should  be  more 
available  in  primary  care  areas  and/or  that  there  was  insufficient  or 
outdated  equipment  in  areas  of  high  patient  concentrations.  Numerous 
commentors  also  cited  the  unavailability  of  unconfirmed  interpretations 
and  the  delay  before  the  confirmed  interpretations  reach  patient  charts. 

E.  SENSITIVITY  ANALYSIS 

In  order  to  test  the  sensitivity  of  the  conclusions  about  the  cost 
effectiveness  of  CAPOC  to  key  variables  and  assumptions ,  several  cases 
were  identified  and  evaluated: 

Case  1:  the  use  of  corpsmen  to  confirm  normal  CAPOC  inter¬ 
pretations,  thus  relieving  physicians  of  overreading 
this  set  of  ECGs, 

Case  2:  exclusion  of  the  ECGs  taken  in  the  intensive  care 
units  at  San  Diego  NRMC  from  the  analysis  because 
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the  process  for  reading  the  ECGs  is  different  from  the 
process  for  the  rest  of  the  ECG  volume. 

Case  3:  consideration  of  the  ECGs  from  the  intensive  care 
units  at  San  Diego  NRMC  as  automatically  meeting 
criteria  for  effective  turnaround  time  and  reader 
because  they  are  read  immediately  by  the  requesting 
physician  and  the  staff  in  these  units  have  training 
in  cardiology,  and 

Case  4:  inclusion  of  a  fixed  rate  of  overhead  cost  in  all 
18  MTFs  for  different  volumes  of  ECGs. 

The  results  of  these  sensitivity  analyses,  contrasted  with  the  cost 
effectiveness  conclusions  from  the  original  analysis,  are  summarized  in 
Table  5-10  and  in  the  following  discussion.  (Note  that  the  sensitivity 
of  results  to  assumptions  about  the  volume  processed  were  examined 
previously  in  Section  B  of  this  chapter.) 

According  to  the  evaluation  criteria  used  to  evaluate  service 
effectiveness  changes  resulting  from  CAPOC,  a  corpsman  becomes  an 
effective  reader  of  ECGs  for  normal  tracings  with  the  assistance  of  the 
computer-generated  interpretation.  Case  1  assumes  that  this  is  adopted  as 
standard  practice  either  at  all  sites  or  at  only  the  major  sites. 

Table  5-10  indicates  that  this  increases  the  cost  effectiveness  of  CAPOC. 
The  change  is  due  to  the  reduction  in  labor  costs  for  confirming  normal 
ECGs  and  a  1.6-minute  physician  time  saving  for  reading  abnormal  ECGs. 

ECGs  taken  in  intensive  care  units  serve  a  different  purpose  than 
other  routine  ECGs  and  the  process  for  taking  and  reading  them  differs 
as  well.  These  ECGs  are  used  to  monitor  the  progress  of  patients  with 
known  or  suspected  myocardial  infarctions  or  other  cardiac  abnormalities. 
At  San  Diego  NRMC  they  are  usually  read  in  the  unit  in  which  they  are 
taken  before  being  sent  to  the  Heart  Station  for  a  formal  reading.  At 
San  Diego,  these  ECGs  accounted  for  18  percent  of  the  X  Period  volume  and 
11  percent  of  the  Y  Period  volume.  If  the  reading  by  the  Heart  Station 
was  counted  as  the  official  reading  for  purposes  of  the  evaluation,  many 
of  these  ECGs  did  not  meet  criteria  for  effective  turnaround  time. 
Therefore,  two  sets  of  different  assumptions  were  examined  with  regard 
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to  these  GCGs.  Case  2  assumes  that  these  ECGs  should  be  excluded  from 
consideration;  variable  costs  of  ECG  services  were  assumed  to  be  reduced 
by  the  same  percentage  as  the  reduction  in  volume.  Case  3  assumes  that 
the  official  reading  occurs  in  the  intensive  care  unit  and  the  entire 
volume  meets  all  criteria  for  preferred  service.  As  shown  in  Table  5-10, 
both  of  these  cases  demonstrate  increased  cost  effectiveness  of  CAPOC. 

In  the  X  Period,  Camp  Pendleton  was  the  only  CAPOC  site  at  which  the 
Uniform  Chart  of  Accounts  had  been  implemented.  Therefore,  this  was  the 
only  site  for  which  data  were  available  on  the  overhead  costs  of  providing 
ECG  services  in  a  MTF.  Because  there  was  no  basis  for  assigning  this 
overhead  rate  to  the  other  sites,  overhead  costs  were  excluded  in  the 
original  cost  analysis.  In  order  to  test  the  sensitivity  of  the  results 
to  the  inclusion  of  overhead  costs,  the  16%  overhead  rate  from  Camp 
Pendleton  was  applied  to  all  of  the  sites.  As  shown  in  Table  5-10,  the 
result  of  this  change  is  to  increase  the  per  ECG  costs  and  the  difference 
in  costs  between  manual  and  automated  services. 

The  sensitivity  analysis  indicates  that  volume  is  the  single  most 
critical  factor  determining  the  cost  effectiveness  of  automated  services. 
It  also  verifies  that  the  original  assumptions  used  were  conservative, 
since  all  of  the  other  assumptions  tested  show  CAPOC  to  be  even  more 
cost  effective. 
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APPENDIX  A 


STAFF  QUESTIONNAIRES  FOR  PRIMARY  EVALUATION  SITES 
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STUDY  OF  ECG  SERVICES 
Wave  1 


For  Computer  Use  Only: 
Re8p.  I.D. _ 

(Col.  1-4) 
Base:  S  -  1  2  3  4  5 

Clinic:  6-1  2  3  4  S 

Date:  7  -Q  2  3 


Please  be  assured  that  we  wish  this  study  to  be  absolutely  anonymous.  We 
can  understand  opinions  best  when  combining  a  person's  answers  to  the  first 
questionnaire  with  his/her  answers  to  the  second  and  third  questionnaires. 
Since  we  don't  want  you  to  write  down  your  name,  we  have  chosen  your  date 
of  birth  as  an  Identifier  to  appear  on  all  your  questionnaires. 

For  example:  a  person  born  on  January  6,  1940,  will  have  an  Identification 
#010640.  We  have  chosen  your  date  of  birth  because  It  Is  a  number  which 
will  not  change  and  no  attempt  will  be  made  to  link  this  Information  with 
your  name. 

Please  write  your  Identifier  here:  _ 

(Identifier) 

This  questionnaire  Is  designed  to  be  keypunched;  for  this  purpose  please 
Indicate  your  answers  by  circling  code  numbers  Instead  of  writing  the 
customary  check  marks.  For  example: 

Are  you  here  to  have  an  ECG  taken?  (circle  code  number)  Yes  S74-T5 

No  -2 


*  *  * 


1.  Which  of  the  following  functions,  if  any,  do  you  perform  related  to  ECGs? 

(Circle  as  many  as  apply) 

Order  ECGs  to  be  taken  on  patients  .  17 -l 

Interpret  ECGs  for  my  own  patients  - 

Interpret  ECGS  for  patients  being  treated  by  others  - 

Check  ECG  interpretations  done  by  others  - - - 

Take  ECGs  . 

File  ECGs  . 

Schedule  patients  for  ECGs - - 

Manage  the  ECG  department  - 

Other  (  specify ) . . 

Do  not  perform  any  functions  related  to  ECGs - - - 

2.  To  your  knowledge,  are  patients  ever  recalled  to  have  a  repeat  ECG  taken  because 
the  technical  quality  of  the  tracing  was  inadequate  for  interpretation? 


Yes  .  18-1 

No .  -2 

Don't  know  — . - . .  -3 
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3.  How  often  would  you  say  these  recalls  occur  due  to  inadequate  EC6  quality? 


V- 


v"  %  i 


ft*:- 


ft-:: 


4. 


8. 


Often - - -  19- 

Seldom  - 

Hardly  ever  - 

Never  - 

Don't  know  - 

To  your  knowledge,  are  patients  ever  recalled  to  have  a  repeat  ECG  taken 
because  their  ECG  has  been  lost? 

Yes . - .  20- 

No . - . - . 

Don't  know  - 

How  often  would  you  say  these  recalls  occur  due  to  the  loss  of  an  ECG? 

Often - 21- 

Seldom  - 

Hardly  ever  - 

Never  - 

Don't  know - - - 

To  your  knowledge,  are  patients  ever  referred  to  other  medical  facilities 
(off  base)  for  cardioloqy  consults? 

Yes . 22- 

No  - 

Don't  know  - 

How  often,  would  you  say,  these  referrals  off  base  occur? 

Often - 23- 

Seldom  - 

Hardly  ever  - 

Never  - 

Don’t  know - - - . 


consul ts?S°me  °f  the  reasons  why  Pat1'ents  are  referred  off  base  for  cardiology 


24 

2b 
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9.  For  each  of  the  following  types  of  locations  where  ECGs  are  taken, 
what  is  an  acceptable  delay  between  the  request  for  a  STAT  ECG  and 
the  time  it  is  taken? 

a.  in  the  emergency  room  (circle  one) 

0-15  minutes - 69- 

16-30  minutes  - 

31-45  minutes  - 

46-60  minutes - — 

over  60  minutes  - 


b.  in  other  medical  treatment  areas  (circle  one) 

0-15  minutes  -  70- 

16-30  minutes  - 

31-45  minutes  - 

46-60  minutes  - 

over  60  minutes  - 

10.  What  is  an  acceptable  delay  between  the  request  for  a  routine  ECG 
and  the  time  it  is  taken?  (circle  one) 

less  than  1  hour - -  71- 

1-4  hours  - 

5-8  hours  - 

9-24  hours  - 

over  24  hours  - 


11. 


12. 


Ideally,  how  long  on  the  average  do  you  think  it  should  take  from  the 
time  a  STAT  ECG  Is  ordered  to  the  time  the  interpretation  is  available? 
(Write  In  the  amount  of  time— minutes  or  hours— in  the  box.) 


Amount  of  time 


Ideally,  how  long  on  the  average  do  you  think  it  should  take  from  the 
time  a  routine  ECG  Is  ordered  to  the  time  the  interpretation  is 
available?  [Write  in  the  amount  of  time— minutes,  hours,  or  days— 
in  the  box.) 


Amount  of  time 


72- 

73- 

74- 

75- 


76- 

77- 

78- 

79- 


END  OF  CARD  80  -© 
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Listed  below  are  attributes  an  ECG  service  can  provide.  Using  a  scale  from 
1  to  5 ,  please  assign  a  rating  to  each  attribute  to  indicate  how  important 
that  service  is  to  you  (e.g.,  a  rating  of  1  means  this  is  very  important  to 
you  and  a  rating  of  5  means  this  service  is  very  unimportant  to  you).  You  can 
choose  any  number  between  l  and  5  and  please  circle  one  number  for  each  line. 

For  example:  If  It  is  very  important  to  you  that  STAT  ECGs  can  be  taken 
24  hours  a  day,  circle  26-®  If  this  is  very  unimportant  to  you,  circle 
26-®  And  if  this  item  strikes  you  as  being  somewhere  in  between  very  impor¬ 
tant,  circle  26- ©or  26-® or  26J&- only  one  circle  for  each  item. 

Very 

Important 

a.  STAT  ECGs  can  be  taken  24  hours  a  day  16 -l  -2 

b.  Routine  ECGs  can  be  taken  24  hours  a  day  17-1  -2 

c.  STAT  ECG  interpretations  are  available  24  hours  a  day  16-1  -2 

d.  Routine  ECG  interpretations  are  available  24  hours  a  day  19-1  -2 

e.  ECGs  can  be  taken  in  all  diagnostic  and  treatment  areas  20~ 1  -2 


time  it  is  taken  Is  within  acceptable  limits* 

Interval  between  the  time  of  a  request  for  a  routi 
the  time  it  is  taken  is  within  acceptable  limits* 


time  the  interpretation  is  available  Is  within  acceptable 
limits* 


the  time  the  Interpretation  is  available  is  within  acceptable 
limits* 

k.  ECG  interpretations  are  accurate 

l.  ECG  interpretations  are  complete  (all  aspects  noted) 

m.  ECG  tracings  are  of  high  technical  quality 

n.  Different  readers  make  the  same  readings 

o.  Interpretations  are  easy  to  read  and  understand 

p.  Prior  ECG  tracings  are  available  for  comparison  purposes 


to  discuss  significance  of  ECG  Interpretation  in  patient 
management 

r.  Other  (epeoify) _ 

*  As  defined  by  you  in  Questions  9-12.  222 


Very 

Unim- 

portar* 


16-1 

-2 

-3 

-4 

-5 

17-1 

-2 

-3 

-4 

-5 

16-1 

-2 

-3 

-4 

-5 

19~1 

-2 

-3 

-4 

-5 

20- 1 

-2 

-3 

-4 

-5 

4m 

21 

-2 

-3 

-4 

-5 

22-1 

-2 

-3 

-4 

-5 

x* 

23^2 

-2 

- 3 

-4 

-5 

u 

24-1 

-2 

-3 

-4 

-5 

7". 

ri 

25-1 

-2 

-3 

-4 

-5 

26-1 

-2 

-3 

-4 

-5 

r* 

27-1 

-2 

-3 

-4 

-5 

28-1 

-2 

-3 

-4 

-5 

*rf 

29-1 

-2 

-3 

-4 

-5 

30-1 

-2 

-3 

-4 

-5 

*■„ 

31-1 

-2 

-3 

-4 

-5 

32-1 

-2 

-3 

-4 

-5 

a 

33-1 

-2 

-3 

-4 

-5 
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These  attributes  are  listed  once  more.  This  time  please  indicate  how  satisfied 
you  are  with  these  attributes  as  provided  by  the  current  ECG  system  here 
(e.g.,  a  rating  of  l  means  always  satisfactory  and  a  rating  of  5  means  never 
satisfactory).  You  can  choose  any  number  between  1  and  5  to  Indicate  how 
satisfied  you  are.  Please  circle  one  code  for  each  line. 

Very 

Un- 

Very  sati 

Satisfied  fied 


Very  Not 

Un-  Appli- 

satis-  cable  or 

fied  Don't  Know 


a.  Number  of  hours  a  day  STAT  ECGs  can  be  taken 

b.  Number  of  hours  a  day  routine  ECGs  can  be  taken 

c.  Number  of  hours  a  day  STAT  ECG  Interpretations  are 
available 

d.  Number  of  hours  a  day  routine  ECG  interpretations  are 
available 

e.  Number  of  diagnostic  and  treatment  areas  where  ECGs 
can  be  taken 

f.  Number  of  diagnostic  and  treatment  areas  where  ECGs 
can  be  interpreted 

g.  Interval  between  the  time  of  a  request  for  STAT  ECG 
and  the  time  it  Is  taken 

h.  Interval  between  the  time  of  a  request  for  a  routine 
ECG  and  the  time  it  is  taken 

i.  Interval  between  the  time  of  a  request  for  a  STAT 
ECG  and  the  time  the  interpretation  is  available 

j.  Interval  between  the  time  of  a  request  for  a  routine 
ECG  and  the  time  the  interpretation  is  available 

k.  Accuracy  of  ECG  interpretations 

l.  Completeness  (all  aspects  noted)  of  ECG  interpre¬ 
tations 

m.  Technical  quality  of  ECG  tracings 

n.  Consistency  of  interpretations  made  by  different 
readers 

o.  Ease  of  reading  and  understanding  Interpretations 

p.  Availability  of  prior  ECG  tracings  for  comparison 
purposes 

q.  Availability  of  experts  (cardiologists  or  other  ECG 
specialists)  to  discuss  significance  of  ECG  interpre¬ 
tation  in  patient  management 


r.  Other  (specify) 


34-1  -2  -3  -4  -5 


35- 1  -2  -3  _ 4 

36- 1  -2  -3  -4 


37-1  -2  -3 


38-1  -2  -3  - 4  -5 


39-1  -2  -3  -4  -5 


40-1  -2  -3 


41-1  -2  -3  -4 


42-1  -2  -3  -4  -5 


43-1  -2  -3  -4  -5 


44- 1  -2  -3  -4  -5 

45- 1  -2  -3  -4  -5 


46- 1  -2  -3  - 4  -5 

47- 1  -2  -3  -4  -5 


48- 1  -2  .3  -4  -5 

49- 1  -2  '-3  -4  -5 


50-1  -2  -3  -4  -5 


51-1  -2  -3  -4  -5 


END  OF  CARD  80 
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-  ,  .  s  v  vs  *: 


15.  Do  you  personally  read/interpret  ECGs  or  overread  ECGs  (i.e.,  check 
the  Interpretation  made  by  someone  else)?  (Circle  as  many  as  apply) 


Skip  to  Q.  20' 
Skip  to  Q.  20- 


Read/lnterpret  ECGs- 

Overread  ECGs  - 

•Do  not  do  either 
-Don't  know  - 


Answer  Q.  16-19  only  If  you  read/interpret  or  overread  ECGs. 


16.  How  many  ECGs  do  you  estimate  you  read  In  an  average  week?  (Write  in  the 
number  of  ECGs  in  the  box  at  the  right) 


Number  of  ECGs 


17.  How  much  time  would  you  say  It  takes  you  on  the  average  to  make  the 
following  types  of  ECG  Interpretations?  Please  write  In  an  answer 
for  each  line.  If  you  don't  interpret  this  type,  circle  the  column 
on  the  right. 


Minutes  Seconds  Don't  Interpret 


a.  Interpretation  of  a  single  ECG  that 
Is  within  normal  limits  -  # 


b.  Interpretation  of  an  abnormal  ECG  —  £_ 


c.  Serial  comparison  with  a  prior  ECG  # 
tracing  - 


d.  Other  Interpretation 


(specify) 


18.  If  you  overread  ECG  interpretations  made  by  others  how  many  do  you 

overread  In  an  average  week?  (Write  in  the  number  of  ECGs  in  the  box). 


Number  of  ECGs 


19.  How  much  time  do  you  spend  overreading  In  an  average  week?  (Write  in  50_ 
the  amount  of  time  in  the  box) 


Amount  of  time 


30- 

31- 

32- 

3 

33- 

34- 

35- 

-•** 

36- 

37- 

r*; 

w  “ 

38- 

39- 

40- 

41- 

42- 

V 

43- 

\s 

44- 

45- 

y 

46- 

47- 

•v't 

48- 

49- 

V.  ‘ 

is 

50- 

51- 

52- 

•V 

53- 
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House  staff  officer  (intern,  resident  or  fellow)  -- 

Nurse  - 

Corpsman  or  technician  - 

Other  (specify) 


21.  How  long  have  you  been  working  in  your  current  job  at  this  facility? 


Less  than  six  months 

6-12  months  - 

13  months  or  longer  ■ 
Don't  know  - 


22.  What  is  your  military  status? 


Active  duty  military 

Civilian  - 

Other ■ (specify) _ 

Don't  know  - 


23.  Please  write  in  any  comments  you  may  have  concerning  the  EC6  system  at  your  base. 


Thank  you  for  your  time  and  effort.  Please  return  this  questionnaire  in  the 
attached  prepaid  self-addressed  envelope. 


Ifc  &l  to  ^  A  fcl  M  M  Cn  Caj 


STUOY  OF  EC6  SERVICES 


\\ 


v 


Wave  2 

In  November  1979  a  computer  system  called  CAPOC  was  installed  to  provide 
computer-assisted  reading  to  electrocardiograms  (ECGs)  at  USAF  Hospital, 
Nellis  Air  Force  Base.  This  Questionnaire  is  designed  to  record  your 
experience  with  the  CAPOC  system,  and  your  views  of  changes  that  have 
occurred  in  EC6  services  at  Nellis,  since  CAPOC  was  installed. 

We  would  like  all  staff  to  answer  questions  about  the  requirements  for 
ECG  services  and  satisfaction  with  current  CAPOC  services. 

If  you  worked  here  before  CAPOC  was  installed,  we  will  ask  you  to  answer  some 
additional  questions  about  changes  in  services. 

Please  be  assured  that  we  wish  your  responses  to  be  anonymous.  For  those 
staff  who  were  here  prior  to  CAPOC  installation,  we  would  like  to  compare 
responses  to  this  questionnaire  with  responses  given  in  the  first  survey 
distributed  a  year  ago.  We  have  chosen  your  date  of  birth  as  an  identifier 
to  appear  on  all  your  questionnaires. 

For  example:  a  person  born  on  January  6,  1940,  will  have  an  identifi¬ 
cation  #010640.  We  have  chosen  your  date  of  birth  because  it  is  a  number 
which  will  not  change.  No  attempt  will  be  made  to  link  this  information 
with  your  name. 

Please  write  your  identifier  here  For  Computer  Use  Only 

_  _ _  _ _  Cols.  (3-8) 

Month  Day  Year  Bases  9-1  2(3)4  5 

Wave  100 

This  questionnaire  is  designed  to  be  keypunched;  for  this  purpose  please 
indicate  your  answers  by  circling  the  code  numbers  (e.g.,  140)  instead 
of  writing  the  customary  check  marks. 


Example: 

Do  your  order  ECGs  to  be  taken  on  your  patients? 


Yes  140 
No  -2 


Arthur  D  Little  Inc 


1.  What  functions,  if  any,  do  you  perform  related  to  ECGs?  (Check  cu> 
many  cu>  apply. } 


a.  Order  ECGs  to  be  taken  on  patients - 11-1 

b.  Interpret  ECGs  for  my  own  patients -  -2 

c.  Interpret  ECGs  for  patients  being  treated  by  others -  -3 

d.  Check  ECG  interpretations  done  by  others -  -4 

e.  Take  ECGs -  -5 

f.  File  ECGs . 6 

g.  Schedule  patients  for  ECGs -  -7 

h.  Manage  the  ECG  department - - -8 

i.  Other  (specify) - -9 

j.  Do  not  perform  any  functions  related  to  ECGs - -0 

la.  Were  you  on  the  staff  at  Nellis  before  October  1979  when 
the  CAPOC  computer-assisted  ECG  system  was  installed? 

Yes . 12-1 

Skip  to  Q.  3  I  No - - - -  -2 

Skip  to  Q.  3  I  Don't  know -  -3 


If  you  were  on  the  staff  of  Nellis  Ajr  Force  Base  prior  to  November  1979  and  have 
experience  with  both  the  manual  and  computerized  ECG  services,  please  answer 
question  lb.  If  not,  proceed  to  question  2. 


lb.  If  the  functions  you  perform  now  differ  from  those  you  performed  before 
CAPOC  was  installed,  please  write  in  what  these  functions  are  and  how 
they  differ  from  what  you  used  to  do  before  CAPOC  installation. 

_ 13- 

_ 14- 


2.  To  your  knowledge,  using  the  CAPOC  system,  are  patients  ever  re¬ 
called  to  have  a  repeat  ECG  taken  because  the  technical  quality 
of  the  tracing  is  inadequate  for  interpretation? 


Yes . 15-1 

No . 2 

Don't  know -  -3 

3.  How  often  would  you  say  these  recalls  occur  due  to  inadequate  ECG 

<,ua,Uy?  Often . 16-1 

Sel  dom -  -2 

Hardly  Ever - - -3 

Never-—- - -4 

Don’t  know— — - -  -5 
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To  your  knowledge,  using  the  CAPOC  system,  are  patients  ever 
recalled  to  have  a  repeat  ECG  taken  because  their  ECG  has  been  lost? 

Yes . 17-1 

No . 2 

Don't  know -  -3 

How  often  would  you  say  these  recalls  occur  due  to  the  loss  of 
an  ECG? 

Often - 18-1 

Seldom -  -2 

Hardly  Evei -  -3 

Never -  -4 

Don't  know -  -5 


If  you  were  on  the  staff  of  Nellis  Air  Force  Base  prior  to  November  1979,  and  have 
experience  with  both  manual  and  computerized  ECG  services  please  answer  questions 
5a.  and  5b.  If  not,  please  proceed  to  question  5c. 

5a.  When  comparing  CAPOC  with  manual  ECG  services  would  you 

say  repeat 

ECGs  taken  because  of  poor  technical  quality  occur: 

More  often  since  CAPOC-—- 

. 19-1 

Less  often  since  CAPOC-— 

. 2 

About  as  often  as  before- 

. 3 

5b.  When  comparing  CAPOC  with  manual  ECG  services  would  you 

say  ECGs 

are  lost:  j 

More  often  since  CAPOC-  — 

. 20-1 

Less  often  since  CAPOC-- 

. 2 

About  as  often  as  before- 

. 3 

5c.  To  your  knowledge,  since  CAPOC  has  been  available,  are  ECGs  ever  taken 
and  interpreted  manually  and  not  transmitted  to  the  computer  system? 


Yes . 

No . 

Don't  know- 


5d.  What  are  some  of  the  reasons,  if  any,  why  a  manual  system  is  used 
instead  of  CAPOC?  (C/iecfe  aa  many  cu>  apply.) 

a.  CAPOC  system  is  not  available  in  all  areas  where  ECGs  are 

taken - 22-1 

b.  CAPOC  system  is  not  available  at  all  times  (e.g.,  not  avail¬ 
able  nights  or  weekends)— -  -2 

c.  CAPOC  system  is  busy -  -3 

d.  CAPOC  system  is  not  working -  -4 

e.  Manual  system  is  more  satisfactory— -  -5 

f.  Other  (Specify) - -6 

g.  Don't  know-—-- - -7 
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24- 


5e.  If  you  Indicated  that  a  manual  system  is  more  satisfactory, 
please  write  in  why  you  say  so. 


5f.  How  frequently  would  you  say,  ECGs  are  taken  manually  and  not 
transmitted  to  the  computer  system? 


Almost  every  day - 25-1 

About  once  a  week -  -2 

About  once  a  month -  -3 

Less  than  once  a  month -  -4 

Never -  -5 

Don't  know -  -6 

6.  To  your  knowledge,  using  the  CAPOC  system,  are  patients  ever 
referred  to  other  medical  facilities  for  cardiology  consults? 

Yes . 26-1 

No . 2 

Don't  know- -  -3 

7.  How  often,  would  you  say,  these  referrals  off  base  occur? 

Often . 27-1 

Seldom -  -2 

Hardly  Ever -  -3 

Never -  -4 

Don't  know - - —  -5 


8.  What  are  some  of  the  reaons  why  patients  are  currently  referred 
off  base  for  cardiology  consults?  [VIzom.  m viltz  .ot.j 

_  28- 

_  29- 


If  you  were  on  the  staff  of  Nellis  prior  to  installation  of  CAPOC  please 
answer  question  8a.  If  not  proceed  to  question  9. 


8a.  Compared  with  the  time  before  CAPOC  was  installed  do  you  feel  the 
frequency  of  off  base  referrals  has  increased,  decreased  or  has  it 


stayed  the  same? 

Increased - 30-1 

Decreased -  -2 

Stayed  the  same -  -3 

Don '  t  know- -  -4 


& 
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9.  For  each  of  the  following  types  of  locations  where  ECGs  are  taken, 
what  is  an  acceptable  delay  between  the  request  for  a  STAT  ECG 
and  the  time  it  is  taken? 

a.  In  the  emergency  room  [cixclz  one) 

0-15  minutes - 31-1 

16-30  minutes -  - 

31-45  minutes -  - 

46-60  minutes -  - 

over  60  minutes -  - 

b.  In  other  medical  treatment  areas  [<Uxc.lt  onz ) 

0-15  minutes - 32- 

16-30  minutes -  - 

31-45  minutes— - - 

46-60  minutes - - - 

over  60  minutes -  - 

10.  What  is  an  acceptable  delay  between  the  request  for  a  routine  ECG 
and  the  time  it  is  taken?  [cixclz  onz ) 

less  than  1  hour - 33- 

1-4  hours—— - — - - - -  - 

5-8  hours- -  - 

9-24  hours — -  - 

over  24  hours -  - 

11.  Ideally,  how  long  on  the  average  do  you  think  it  should  take  from 
the  time  a  STAT  ECG  is  ordered  to  the  time  the  interpretation  is 
available?  [lilxitz  In  thz  amount  otf  timz  and  Indicatz  mcmjutz6  ox 
houxi  In  thz  box. ) 


Amount  of  time- 


(34  -  37) 


12,  Ideally,  how  long  on  the  average  do  you  think  it  should  take  from 
the  time  a  routine  ECG  is  ordered  to  the  time  the  interpretation 
is  available?  Wxitz  in  thz  amount  o£  timz — minute* ,  houx*,  ox 
day* --in  thz  box. ) 

Amount  of  time 
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Listed  below  are  attributes  an  ECG  service  can  provide.  Using  a  scale  from 

1  to  5,  please  assign  a  rating  to  each  attribute  to  indicate  how  important 

that  service  is  to  you  (e.g. ,  a  rating  of  1  means  this  is  very  important  to 

you  and  a  rating  of  5  means  this  service  is  very  unimportant  to  you).  You 

can  choose  any  number  between  1  and  5  and  please  ci rcle  one  number  for  each  f 

line. 


For  example;  If  it  is  very  important  to  you  that  STAT  ECGs  can  be  taken 
24  hours  a  day,  circle  42(p.  If  this  is  very  unimportant  to  you,  circle 
42-OjIl  And  if  this  item  strikes  you  as  being  somewhere  in  between  very 
important  and  very  unimportant,  circle  42(|)or  42@or  42(3)—  only  one 
circle  for  each  item. 


Very 


Very 

Unim- 

Important 

portant 

„ 

a. 

STAT  ECGs  can  be  taken  24  hours  a  day 

42-1 

-2 

-3 

-4 

-5 

_ 

b. 

Routine  ECGs  can  be  taken  24  hours  a  day 

43-1 

-2 

-3 

-4 

-5 

c. 

STAT  ECG  interpretations  are  available  24  hours  a 

44-1 

-2 

-3 

-4 

-5 

S"# 

day 

*  - 

d. 

Routine  ECG  interpretations  are  available  24  hours 
a  day 

45-1 

-2 

-3 

-4 

-5 

e. 

ECGs  can  be  taken  in  all  diagnostic  and  treatment 
areas 

46-1 

-2 

-3 

-4 

-5 

f. 

ECGs  can  be  interpreted  in  all  diagnostic  and 
treatment  areas 

47-1 

-2 

-3 

-4 

-5 

g- 

Interval  between  the  time  of  a  request  for  a  STAT 

48-1 

-2 

-3 

-4 

-5 

% 

ECG  and  time  it  is  taken  is  within  acceptable  limits* 

h. 

Interval  between  the  time  of  a  request  for  routine 

49-1 

-2 

-3 

-4 

-5 

ECG  and  the  time  it  is  taken  is  within  acceptable 

1 imi ts 

~  * 

i . 

Interval  between  the  time  of  a  request  for  a  STAT 

50-1 

-2 

-3 

-4 

-5 

*• 

ECG  and  the  time  the  interpretation  is  available 
is  within  acceptable  limits* 

. 

j. 

Interval  between  the  time  of  a  request  for  a  rou- 

51-1 

-2 

-3 

-4 

-5 

V* 

tine  ECG  and  the  time  the  interpretation  is  avail¬ 
able  is  within  acceptable  limits* 

k. 

ECG  interpretations  are  accurate 

52-1 

-2 

-3 

-4 

-5 

1. 

ECG  interpretations  are  complete  (all  aspects 
noted) 

53-1 

-2 

-3 

-4 

-5 

m. 

ECG  tracings  are  of  high  technical  quality 

54-1 

-2 

-3 

-4 

-5 

■j 

n. 

Different  readers  make  the  same  readings 

55-1 

-2 

-3 

-4 

-5 

0. 

Interpretations  are  easy  to  read  and  understand 

56-1 

-2 

-3 

-4 

-5 

P- 

Prior  ECG  tracings  are  available  for  comparison 
purposes 

57-1 

-2 

-3 

-4 

-5 

q. 

Experts  (cardiologists  or  other  ECG  specialists) 
are  available  to  discuss  significance  of  ECG 

58-1 

-2 

-3 

-4 

-5 

*1 

interpretation  in  patient  management 

*«. 

r. 

Other  (specify) 

59-1 

-2 

-3 

-4 

-5 

*As  defined  by  you  in  Questions  9-12. 
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14.  These  attributes  are  listed  once  more.  This  time  please  Indicate  how  satisfied 
you  are  with  these  attributes  as  provided  by  the  current  CAPOC  ECG  system. 

(e.g. ,  a  rating  of  1  means  always  satisfactory  and  a  rating  of  5  means  never 
satisfactory).  You  can  choose  any  number  between  1  and  5  to  indicate  how 
satisfied  you  are.  Please  circle  one  code  for  each  line. 


Very 

Satisfied 

Very  Un¬ 
satisfied 

Not  Appli¬ 
cable  or 
Don-t  know 

a. 

Number  of  hours  a  day  STAT  ECGs  can  be  taken 

60-1 

-2  -3  -4 

-5 

-6 

b. 

Nurrfcer  of  hours  a  day  routine  ECGs  can  be 
taken 

61-1 

-2  -3  -4 

-5 

-6 

c. 

Number  of  hours  a  day  STAT  ECG  interpre¬ 
tations  are  available 

62-1 

-2  -3  -4 

-5 

-6 

d. 

Number  of  hours  a  day  routine  ECG  interpre¬ 
tations  are  avai lable 

63-1 

-2  -3  -4 

-5 

-6 

e. 

Number  of  diagnostic  and  treatment  areas 
where  ECGs  can  be  taken 

64-1 

-2  -3  -4 

-5 

-6 

f. 

Number  of  diagnostic  and  treatment  areas 
where  ECGs  can  be  interpreted 

65-1 

-2  -3  -4 

-5 

-6 

g- 

Interval  between  the  time  of  a  request  for 
a  STAT  ECG  and  the  time  it  is  taken 

66-1 

-2  -3  -4 

-5 

-6 

h. 

Interval  between  the  time  of  a  request  for  a 
routine  ECG  and  the  time  it  is  taken 

67-1 

-2  -3  -4 

-5 

-6 

i . 

Interval  between  the  time  of  a  request  for  a 
STAT  ECG  and  the  time  the  interpretation  is 
available 

68-1 

-2  -3  -4 

-5 

-6 

j. 

Interval  between  the  time  of  a  request  for  a 
routine  ECG  and  the  time  the  interpretation 
is  available 

69-1 

-2  -3  -4 

-5 

-6 

k. 

Accuracy  of  ECG  interpretations 

70-1 

-2  -3  -4 

-5 

-6 

1. 

Completeness  (all  aspects  noted)  of  ECG  in¬ 
terpretations 

71-1 

-2  -3  -4 

-5 

-6 

m. 

Technical  quality  of  ECG  tracings 

72-1 

-2  -3  -4 

-5 

-6 

n. 

Consistency  of  Interpretations  made  by  dif¬ 
ferent  readers 

73-1 

-2  -3  -4 

-5 

-6 

0. 

Ease  of  reading  and  understanding  interpre¬ 
tations 

74-1 

-2  -3  -4 

-5 

-6 

P- 

Availability  of  prior  ECG  tracings  for  com¬ 
parison  purposes 

75-1 

-2  -3  -4 

-5 

-6 

Availability  of  experts  (cardiologists  or 
other  ECG  specialists)  to  discuss  si gni fl¬ 
ea  of  ECG  interpretation  in  patient 

maragement 

76-1 

-2  -3  -4 

-5 

-6 

r. 

Other  (specify) 

77-1 

-2  -3  -4 

-5 

-6 

END  OF  CARD  S0<T) 
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14a.  If  you  worked  at  this  hospital  before  November  1979  when  CAPOC  was  installed, 
please  indicate  if  ECG  services  with  CAPOC  have  changed  for  the  better 
or  the  worse,  or  if  no  change  has  taken  place.  If  you  were  not  here  prior 
to  CAPOC,  proceed  to  question  14b. 


Changed 

Changed 

Have 

for  the 

for  the 

Not 

Don  1 

Better 

Worse 

Chanqed 

Know 

a. 

Number  of  hours  a  day  STAT  ECGs  can  be  taken 

11-1 

-2 

-3 

-4 

b. 

Number  of  hours  a  day  routine  ECGs  can  be  taken 

12-1 

-2 

-3 

-4 

c. 

Number  of  hours  a  day  STAT  ECG  interpretations 
are  available 

13-1 

-2 

-3 

-4 

d. 

Number  of  hours  a  day  routine  ECG  interpreta¬ 
tions  are  available 

14-1 

-2 

-3 

-4 

e. 

Number  of  diagnostic  and  treatment  areas  where 
ECGs  can  be  taken 

15-1 

-2 

-3 

-4 

f. 

Number  of  diagnostic  and  treatment  areas  where 
ECGs  can  be  interpreted 

16-1 

-2 

«■  3 

-4 

g- 

Interval  between  the  time  of  a  request  for  STAT 
ECG  and  the  time  it  is  taken 

17-1 

-2 

-3 

-4 

h. 

Interval  between  the  time  of  a  request  for  a 
routine  ECG  and  the  time  it  is  taken 

18-1 

-2 

-3 

-4 

i. 

Interval  between  the  time  of  a  request  for  a 

STAT  ECG  and  the  time  the  interpretation  is 
available 

19-1 

-2 

-3 

-4 

j. 

Interval  between  the  time  of  a  request  for  a 
routine  ECG  and  the  time  the  interpretation  is 
available 

20-1 

-2 

-3 

-4 

k. 

Accuracy  of  ECG  interpretations 

21-1 

-2 

-3 

-4 

1. 

Completeness  (all  aspects  noted)  of  ECG  inter¬ 
pretations 

22-1 

-2 

-3 

-4 

m. 

Technical  quality  of  ECG  tracings 

23-1 

-2 

-3 

-4 

n. 

Consistency  of  interpretations  made  by  different 
readers 

24-1 

-2 

-3 

-4 

0. 

Ease  of  reading  and  understanding  interpreta¬ 
tions 

25-1 

-2 

-3 

-4 

P- 

Availability  of  prior  ECG  tracings  for  compari¬ 
son  purposes 

26-1 

-2 

-3 

-4 

q- 

Availability  of  experts  (cardiologists  or  other 
ECG  specialists)  to  discuss  significance  of  ECG 
interpretation  In  patient  management 

27-1 

-2 

-3 

-4 

r. 

Other  (specify) 

28-1 

-2 

-3 

-4 
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14b.  In  general,  how  satisfied  are  you  with  the  CAPOC  system  at  this 
base?  Would  you  say  you  are: 


14c.  Why  do  you  say  so? 


Extremely  satisfied- 

Very  satisfied - 

Fairly  satisfied - 

Somewhat  unsatisfied 
Very  unsatisfied-— 
Don't  know - 


Do  you  personally  read/interpret  ECGs  or  overread  ECGs  (i.e., 
check  the  interpretation  made  by  someone  else)?  [ClncJit  cu>  many 
cu>  apply) 

Read/interpret  ECGs - 

Overread  ECGs— - 

Skip  to  Q.  2Ck - Do  not  do  either - 

Skip  to  Q.  2Q< - Don't  know - 


Answer  Q.  16  -  Q.  19e.  only  if  you  read/interpret  or  check  ECGs.  If  you  do 
not  read  ECGs  proceed  to  question  20. 


16.  How  many  ECGs  do  you  estimate  you  read  in  an  average  week?  (Unite 
In  the  numben  o<j  ECGa  In  the.  box  at  the.  night) 


Number  of  ECGs— 


17.  How  much  time  would  you  say  it  takes  you  on  the  average  to  make  the 
following  types  of  ECG  interpretations?  Please  write  in  an  answer 
for  each  line.  If  you  don't  Interpret  this  type,  circle  the  column 
on  the  right.  Don't 

Minutes  Seconds  Interpret 

a.  Interpretation  of  a  single  ECG 

that  is  within  normal  limits- — - —  #  #  37-1 


b.  Interpretation  of  an  abnormal  ECG —  # 


c.  Serial  comparison  with  a  prior  ECG 
traci  i 


d.  Other  Interpretation 


[specify; 
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18.  How  lany  ECG  interpretations  performed  by  others  do  you  check  in 

an  average  week?  (Waite  in  the  numbea  o ^  ECGa  in  the  box  at  the 
xigkt)  - 

Number  of  ECGs - —  _ 

19.  How  much  time  do  you  spend  doing  this  in  an  average  week?  [Waite 
in  the  amount  o{,  time  in  the  box  at  the  night) 


Amount  of  Time- 


60  -  63 ' 


19a.  with  the  current  CAPOC  system,  how  accurate  do  you  feel  the  ECG 
interpretations  used  in  the  patient  care  are  for  the  following 


di agnoses 


d.  Rhythm  abnormalities--- 

e.  Myocardial  infarctions- 


Hypertrophy- 


Always 

Usual ly 

Often 

Don1 

Accurate 

Accurate 

Accurate 

Know 

-64-1 

-2 

-3 

-4 

-65-1 

-2 

-3 

-4 

-66-1 

-2 

-3 

-4 

-67-1 

-2 

-3 

-4 

-68-1 

-2 

-3 

-4 

-69-1 

-2 

-3 

-4 

-70-1 

-2 

-3 

-4 

-71-1 

-2 

-3 

-4 

If  you  worked  at  Nellis  prior  to  November  1979  and  are  familiar  with  both 

the  current  CAPOC  services  and  previous  manual  ECG  services,  please  answer  question 

19b.  If  not  proceed  to  question  19c. 


19b.  When  comparing  computer-assisted  readings  with  the  previous  manual 
reading,  which  method,  in  your  opinion,  provides  better  accuracy 
for  the  following  types  of  ECGs? 


CAPOC 


Manual 
sys tern 


more 

more 

Di  ffer- 

Don't 

Accurate 

Accurate 

ence 

Know 

a. 

All  ECGs . 

. 72-1 

-2 

-3 

-4 

b. 

Normal  ECGs - 

. 73-1 

-2 

-3 

-4 

c. 

All  abnormal  ECGs - 

. 74-1 

-2 

-3 

-4 

d. 

Rhythm  abnormalities - 

. 75-1 

-2 

-3 

-4 

e. 

Myocardial  infarctions - 

. 76-1 

-2 

-3 

-4 

f. 

ST-T  wave  abnormalities - 

. 77-1 

-2 

-3 

-4 

9- 

Hypertrophy - 

. 78-1 

-2 

-3 

-4 

h. 

Conduction  abnormalities- - 

. 79-1 

-2 

-3 

-4 

END  OF  CARD  80-© 


19c.  If  the  computer  makes  errors  In  reading  are  these  generally  false 
positive  or  false  negative  errors? 

False  positive  errors . . — 1 1-1 

False  negative  errors -  -2 

Don't  know -  -3 

19d.  Since  CAPOC  was  installed,  do  you  think  the  amount  of  time  you 
spend  each  week  reading/interpreting  ECGs  has: 

Increased - 1 

Decreased - 

Remained  the  same - 

Other  (specify)  _ — 

Don't  know - 

196.  Why  do  you  say  so?  {w/vLtz  in] 


uestions 


20.  What  is  your  professional  specialty? 

15-1 


21.  How  long  have  you  been  working  in  your  current  job  at  this  facility 

Less  than  six  months— - 16- 

6-12  months -  - 

13  months  or  longer -  - 

Don't  know— -  - 

22,  What  is  your  military  status? 

Active  duty  military - 17- 

Civilian -  - 

Other - - -  - 

Don't  know -  - 


Staff  physician  (specialty) _ _ 

House  staff  officer  (intern,  resident  or 

Nurse - 

Corpsman  or  technician- - 

Other  (specify) 
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csjro^j-Lfi  o-  i —  t\j  co  *3-  , — 


23.  Would  you  like  to  write  in  any  impressions  or  opinions  you  have 
about  the  ECG  system  at  this  facility? 


18- 


19- 

20- 
21- 


END  OF  CARD  80-® 


Thank  you  for  your  time  and  effort.  Please  return  this  questionnai re  in 
the  attached  envelope  to: 


Maj.  T.  Habl itzel/SGHP 
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Site  Visit-Interview  Guide 
CAPOC  Sites 


Facility  Name: 

Address : 

Affiliation  (?  part  of  regional  network): 

Describe  Area  (urban,  rural,  remote,  geography):  • 

Usual  Referral  Centers?  For  Cardiac  Problems? 

CAPOC  Representative: 

Phone  if: 

Other  Persons  Contacted  (name,  position): 

I.  Basic  Site  Information  (obtain  from  CAPOC  rep.) 

Type  of  Facility:  regional  hospital,  local  hospital,  clinic 

Number  of  Physicians  _ ;  Cardiologists  _ ;  IM _ 

Inpatient:  Number  of  beds  _ ;  %  occupancy  _ %;  admissions/year 

Special  Clinical  Units:  CCU,  ICU,  ER  etc.  (Bize  of  each?) 

Outpatient :  Outpatient  visits/year  _  (by  department?) 

Cardiology  clinic?  Y/N  Internal  Medicine  Y/N 

•  Describe  type  of  clinic,  frequency,  patient  population  etc. : 

Comments 
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Continued 


Date  of  CAPOC  implementation? 


Has  CAPOC  system  been  working  well?  What  problems  have  they  had? 


What  is  the  turnaround  time  for  different  types  of  ECGs?  STAT, 
pre-op,  ER,  routine?  Probe  for  supporting  evidence. 


How  satisfied  have  they  been  with  the  manual  system? 


What  advantages  and  disadvantages  do  they  forsee  with  CAPOC? 

Possible  advantages  are:  decreased  turnaround  time;  improved  tracing 
quality;  corpsman  can  identify  normal /abnormal  interpretations;  better 
record  storage. 


Possible  disadvantages  are:  more  work  required;  increased  turnaround 
time  for  some  ECGs. 
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Number  of  Carts 


//3-channel  transmitting  _  #l-channel  _ 

Location  of  carts  by  type  (e.g.,  3-channel-ICU) : 

•  Mobile  cart  or  dedicated  to  the  location? 

•  Age  of  carts: 


How/where  are  ECGs  filed?  How  long  kept? 


List  equipment  acquired  for  CAPOC: 
printer 

ECG  chart  recorder 
CRT  for  editing 
new  phone  lines 


What  will  (or  has  happened)  to  old  ECG  equipment? 
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List  equipment  acquired  for  CAPOC: 

printer 

ECG  chart  recorder 
CRT  for  editing 
new  phone  lines 
ECG  cost  number 


What  will  (or  has  happened  to  old  ECG  equipment? 
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II.  ECG  Information 


a.  Technician  Interview 

Persons  Contacted: 


Number  of  Full-Time  Technicians  _ 

Number  Part  Time  _  Estimate  %  of  Time  on  ECGs  _ % 

Estimate  Total  Full-Time  Equivalent  Technicians  _ 

(e.g.,  2  full-time  plus  1  at  50%  time  =  2.5  FTE) 

Do  the  technicians  plan  to  be  working  at  this  site  through  1980? 

Is  there  a  "chief  technician"?  Name:  _ 

Can  we  call  him/her  by  phone  for  more  follow-up  questions  later?  Y/N 
Phone//:  _ 

What  days/hours  do  they  take  ECGs?  (e.g.,  M -  F/800-1700) 

How  many  ECG's  taken  1  day?  _  Variation  in  daily  volume? 

•  What  happens  when  there  is  no  coverage  on  nights  and  weekends 
(if  applicable)? 


How  much  time  is  spent  on  clerical  duties  e.g.,  typing,  filing, 
etc.?  _  hours/per  day 


Is  one  individual  responsible  for  these  activities  or  do  they  share  it? 


How  often  are  old  ECG's  pulled  for  serial  comparisons? 
What  %  of  ECG's  get  serial  comparisons? 
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b.  Describe  ECG  process  (example  given  below).  Identify  how  process 
varies  by  type  of  patient  or  by  purpose  of  exam  (routine,  STAT, 
pre-op,  ER,  annual  physical,  pediatrics).  Are  pre-ops  called  in 
or  hand-delivered? 


Hypothetical  ECG  Process 


1.  Order  ECG 

2.  Forward  order  to  Lab 

3.  Sort  Request 

4.  Send  technician  to  take  ECG/patient  comes  to  lab 

5.  Prepare  patient 

6.  Record  Tracing 

7.  Record  patient  information 

8.  Label  tracing 

9.  Go  to  next  patient  and  repeat  5-8 

10.  Mount  ECG's  for  reading 

11.  Obtain  old  tracing  for  comparison 

12.  Send  to  reader 

13.  Read 

14.  Type  interpretation 

15.  Copy  report 

16.  Send  ECG  report  to  floor/bring  to  mail  room 

17.  Store  ECG 


Estimate  turnaround  time  for  parts  of  process  and  total  process  where  available 
for  those  reading  role  referred. 
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III.  Reader  Interview 

Number  of  physicians  reading _ 

No.  of  readings  per  day-inpatient  Outpatient  Annual  Physical 
(Probe  for  confirmation  of  previous  tables.) 

Specialty  of  Readers: 

Cardiologist _ 

Internal  Medicine _ 

Other  (specify) 

Estimate  volume,  or  %,  of  ECG  read  by  each. 

Is  there  a  usual  procedure  or  policy  for  dividing  up  reading  workload? 
(E.g.,  does  cardiologist  read  all  the  complex  tracings  by  referral 
from  the  other  readers?) 


What  is  the  reading  schedule? 

How  much  time  is  spent  in  reading? 

Do  they  overread  for  other  sites? 

Is  interpretation  handwritten  or  typed? 

When  are  Pre-ops  and  STATs  read? 

Serial  comparison: 

X  of  ECGs  with  serial  comparison? _ 

X  where  old  ECG  is  available? _ 

Under  what  conditions  is  serial  comparison  done? 

How  often  are  patients  referred  for  cardiac  problems? 

X  triggered  by  ECG? _  Where  referred? 

Were  they  satisfied  with  manual  system? 

Are  they  satisfied  with  CAPOC? 

What  problems  have  they  observed? 
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III.  Reader  Interview  (con't) 

What  do  they  see  as  changes  or  potential  changes  due  to  CAPOC? 

•  Change  in  reading  quality? 

•  Change  in  turnaround  time? 

•  Improved  tracing  quality? 

•  Time  spent  reading? 

•  Number  and  type  of  ECGs  sent  out  for  reading? 

•  Are  non-confirmed  computer  interpretations  used  in  decisions  on 

patient  care? 

•  Availability  of  old  tracings  for  serial  comparisons  due  to  automated 

storage  and  retrieval? 
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4  k.'b 


ill 


IV.  Interview  with  MD  or  Nurse  Users  (i.e.,  non-readers) 

Persons  Contacted: 


What  is  role  with  ECGs? 

Are  they  aware  of  CAPOC?  Describe. 

What  problems  do  they  see  (or  have  they  seen)  with  the  manual  system? 
Have  they  seen  any  improvements  or  problems  due  to  CAPOC? 

What  would  they  like  to  see  changed? 

Do  they  use  the  non-confirmed  computer  interpretation  in  patient  care? 
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V.  Cost  Information 


Obtain  costs  on  the  following  items: 

Labor  $  /yr 

Supplies 

ECG  equipment  maintenance 
ECG  equipment  repair 
Amortization  of  ECG  carts 
Overhead 


& 
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IV.  Interview  with  MD  or  Nurse  Users  (i.e.,  non-readers) 

Persons  Contacted: 


What  is  role  with  ECGs? 

Are  they  aware  of  CAPOC?  Describe. 

What  problems  do  they  see  (or  have  they  seen)  with  the  manual  system? 
Have  they  seen  any  improvements  or  problems  due  to  CAPOC? 

What  would  they  like  to  see  changed? 

Do  they  use  the  non-confirmed  computer  interpretation  in  patient  care? 
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V.  Cost  Information 


Obtain  costs  on  the  following  Items: 

Labor 

Supplies 

ECG  equipment  maintenance 
ECG  equipment  repair 
Amortization  of  ECG  carts 
Overhead 


« 


$ 


/yr 


*■ 

V. 
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Continued 


ECG  Volumes  for  the  period  10/1/78  through  9/30/79 


•  By  location 


Central  Station  Clinic  Emergency  Room 

October,  1978 

November 

December 

January,  1979 

February 

March 

April 

May 

June 

July 

August 

September 
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a.  Continued 


•  By  patient  type 

Out-  Annual 

Inpatient  Routine  STAT  Pre-Op  patient  ER  Phys ■ 

October,  1978 

November 

December 

January,  1979 

February 

March 

April 

May 

June 

July 

August 

September 


What  are  the  approximate  percentage  breakdowns  of  normal  and  abnormal 
ECGs  for: 

abnormal  normal 

inpatients 
outpatients 
annual  physical 
emergency  room 
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CAPOC  Y  Period 


Site  Visit — Interview  Guide 


Date  of  Visit: 

Facility  Name: 


CAPOC  Representative: 

Other  Persons  Contacted  (include  rank,  position,  etc.) 


I.  Basic  Site  Information 

a.  Number  of  Physicians  by  type  _ # 

Cardiologists  _ 

Internists  _ 

GP  _ 

Flight  Surgeon  _ 

Other  (or  comments) 


b.  Inpatient  Information: 

Number  of  beds  _ ;  %  occupancy _ 

Admissions  per  year  _ _ _ _ ____________ 

Have  there  been  any  major  changes  in  inpatient  care  during 
1980  (e.g.,  cardiologist  left  staff,  ER  closed)? 


c.  Outpatient  Information: 

Outpatient  visits  per  year  _ _ _ _ 

Have  there  been  any  major  changes  in  outpatient  services? 
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II.  CAFOC  Information 


Date  of  Implementation: 


CAFOC  Cart(s) : 


Location 


Confirm  Other  Equipment: 

DEC  Writer  _  Other  Printer 

ECG  Writer  CRT 


New  Phone  Lines  _ 

(overread  center  equipment) 


Major  Problems  with  CAPOC?  (e.g.,  carts,  transmitting  problems, 
communication  lines,  service,  training,  lost  ECGs) 

Probe  for  specific  examples: 


III.  Equipment  Information 

Confirm  equipment  list  (see  attached  sheets).  If  applicable, 
find  out  what  happened  to  ECG  machine (s)  replaced  by  CAPOC. 
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IV.  Benefits 


For  each  type  of  ECG,  find  out  how  CAPOC  has  impacted  turnaround 
time,  reader,  serial  comparison.  (Use  attached  table) 


Confirm  types  of  ECGs,  percent  of  each  type,  mix  of  normal/abnormal, 
readers  for  each  type. 


Comments : 
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Turnaround  Time 


Did  CAPOC  replace  mail  for  any  part  of  ECG  process? 


How  long  does  it  take  to  transmit?  To  receive  unconfirmed 
interpretation?  Confirmed  interpretation? 


Do  ECGs  get  to  physicians  for  reading  more  quickly? 


Do  physicians  read  ECGs  more  promptly? 


Is  change  in  turnaround  time  sufficient  to  change  %  of  ECGs 
meeting  criteria? 


ACCEPTABLE 

OPT  INAL  ; 

1ST  AT  I  ENT  ECGs  | 

toutin* 

7  uays 

1  day  | 

"GUT" 

30  min. 

1 

15  aln.  ; 

Pre- Operative 

24  hours* 

12  nours*  ! 

Eatrgency  Hoorn 

30  nln. 

15  Bln . 

OUTPATIENT  ECGs 

Diagnostic  (for  that  visit) 

30  Bin. 

30  Bln. 

(for  future  visit) 

3  days 

30  Bin. 

Other  Screening 

7  days 

2  days 
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Are  some  physicians  reading  ECGs? 


Has  reading  schedule  changed? 


Has  amount  of  time  spent  on  reading  changed? 
For  normals?  For  abnormals? 


Is  unconfirmed  interpretation  used  in  patient  care? 


Is  confirmed  interpretation  edited  in  system  or  handwritten  on 
unconfirmed  report? 


Are  pre-op  or  STAT/ER  ECGs  handled  differently? 


Does  CAPOC  help  in  making  better  interpretations? 
More  consistent  interpretations? 


Does  CAPOC  interpret  normals  correctly? 


Does  CAPOC  interpret  abnormals  correctly? 

Does  it  overcall  (false-positive)  or  undercall  (false-negative)  the  following 
Arrhythmia  MI  LVH  ST-T  waves 


Do  corpsmen  or  physicians'  assistants  confirm  computer-interpreted  normals? 
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Serial  Comparison 

Has  %  of  ECGs  receiving  serial  comparison  changed? 

Has  file/record  system  changed  due  to  CAPOC? 

Do  they  rely  on  CAPOC  system  for  all  files? 

Have  they  noticed  (or  do  they  anticipate)  more  old  records  retrieved 
due  to  CAPOC? 


Other  Benefits 

Has  quality  of  tracings  improved? 

Have  referrals  increased  or  decreased?  (Can  specific  cases/patients 
be  remembered?) 
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Have  costs  changed? 


Technician  Labor  Costs 


Do  technicians  spend  more  or  less  time  taking  ECGs?  processing 
ECGs?  filing  ECGs?  typing /editing  ECGs? 


If  they  save  time,  how  do  they  spend  it? 


What  percent  of  time  is  spent  on  ECGs?  How  many  FTE's? 


Are  labor  costs  for  ECGs  recorded?  Is  salary  information  available? 


Physician  Labor  Costs 


Does  reading  take  less  time  for  normals?  Abnormals? 


How  long  do  physicians  spend  reading?  Per  ECG?  Overall? 
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Supplies  Cost 


Cost  of:  paper,  gauze,  electrodes,  gel,  plastic  cover  sheets? 


Maintenance  &  Repair  Costs 


Capital  Costs 


Confirm  equipment  table 


How  do  they  amortize  equipment,  if  applicable? 
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VI.  General 


Obtain  ECG  Volume  by  type  of  ECG  for  September  1979 
through  October  1980. 


Indicate  which  ECGs  are  being  done  by  CAPOC. 


Describe  process  or  confirm  manual  process  as  unchanged. 
Understand  differences  by  type  of  ECG. 
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